{"id":6675,"date":"2021-02-15T02:37:29","date_gmt":"2021-02-15T02:37:29","guid":{"rendered":"https:\/\/www.insightmind.com.my\/site\/?p=6675"},"modified":"2021-02-18T04:10:08","modified_gmt":"2021-02-18T04:10:08","slug":"ghostly-whitespaces-that-spooks-data-scientists","status":"publish","type":"post","link":"https:\/\/www.insightmind.com.my\/site\/ghostly-whitespaces-that-spooks-data-scientists\/","title":{"rendered":"Ghostly whitespaces that spooks data scientists and what can save their day"},"content":{"rendered":"[vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_shadow=”none” column_border_radius=”none” column_link_target=”_self” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” tablet_text_alignment=”default” phone_text_alignment=”default” column_border_width=”none” column_border_style=”solid” bg_image_animation=”none”][vc_column_text]\r\n
Ghostly whitespaces that spooks data scientists and what can save their day.<\/p>\r\n
Data scientists often wonders if whitespaces in their codes and data would cause some problems and whether it makes sense to replace them with \u201c_\u201d or \u201c-\u201c instead.\u00a0 While whitespaces exist out of necessity in Python codes as tab indentation in functions and for loops, its presence in data can be troublesome. Let\u2019s confirm if Pandas column name for example can tolerate whitespace.<\/p>\r\n
<\/p>\r\n[\/vc_column_text][vc_column_text]\r\n
Data Scientist Whitespaces Example 1:<\/u><\/strong><\/h3>\r\n
The first column name, \u2018Data Engineering\u2019, is separated by whitespace. When we run the following in Jupyter Notebook, we get the desired output \u2013 confirming that whitespace is tolerated in column name.<\/p>\r\n[\/vc_column_text][vc_column_text]<\/p>\r\n<\/figure>\r\n[\/vc_column_text][vc_column_text]\r\n
Now let\u2019s check if whitespace in list items and dataframe column values is giving us any problem. Whitespace separated \u2018data cleansing\u2019 as a new entry in \u2018Data Engineering\u2019 column would serve the purpose. Once again, we get the desired output when run in Jupyter Notebook without any error.<\/p>\r\n\r\n<\/figure>\r\n\r\n
But now can the same seemingly innocent whitespaces in our data giving as nightmare when we process them further for, say visualization purposes.<\/p>\r\n
<\/p>\r\n[\/vc_column_text][vc_column_text]\r\n
Data Scientist Whitespaces Example 2:<\/u><\/strong><\/h3>\r\n
Let\u2019s see another example where we need to produce pivot table before plotting a stacked bar chart sums of the counts of keywords.<\/p>\r\n<\/figure>\r\n
When we run the following codes and inspect the dataframe it gives the desired output along with correct sums of keywords. From the pivot table we do not notice any whitespace that worries us since our earlier observation suggests that nothing should go wrong. But unfortunately, when we start creating traces for our stacked bar chart, and assuming there are no problematic whitespaces, we get key error.<\/p>\r\n<\/figure>\r\n
<\/p>\r\n[\/vc_column_text][vc_column_text]\r\n
The Solution<\/u><\/strong><\/h3>\r\n
It says something is wrong with our keyword names \u2018DE\u2019 and \u2018Python\u2019. We could not have guessed that this is a whitespace problem.There are whitespaces in our keywords and we are calling them without the whitespaces. Let\u2019s now see if Pandas can reveal this whitespace problem, and the following codes expose it. Obviously, we can see trailing whitespace after both keywords.<\/p>\r\n<\/figure>\r\n
Fortunately a simple Lambda function can be employed to rename our column names and simultaneously stripping the trailing whitespaces.<\/p>\r\n<\/figure>\r\n
![\"Data](\"https:\/\/www.insightmind.com.my\/site\/wp-content\/uploads\/2021\/02\/whitespaceCode-1.png\")
<\/figure>\r\n[\/vc_column_text][vc_column_text]\r\n![\"Data](\"https:\/\/www.insightmind.com.my\/site\/wp-content\/uploads\/2021\/02\/whitespaceCode2.jpg\")
<\/figure>\r\n\r\n![\"whitespaceCode3\"](\"https:\/\/www.insightmind.com.my\/site\/wp-content\/uploads\/2021\/02\/whitespaceCode3.jpg\")
<\/figure>\r\n![\"whitespaceCode4\"](\"https:\/\/www.insightmind.com.my\/site\/wp-content\/uploads\/2021\/02\/whitespaceCode4.jpg\")
<\/figure>\r\n![\"Data](\"https:\/\/www.insightmind.com.my\/site\/wp-content\/uploads\/2021\/02\/whitespaceCode5.jpg\")
<\/figure>\r\n![\"Data](\"https:\/\/www.insightmind.com.my\/site\/wp-content\/uploads\/2021\/02\/whitespaceCode6.jpg\")
<\/figure>\r\n