ah2polivio-fls.ipynb

{"cells":[{"metadata":{},"id":"97216d3b","cell_type":"markdown","source":"**This is just an introduction to the new ds4n6_lib, showing some of the analysis we can perform with just a few instructions. During the next weeks we will be publishing different blog posts explaining new techniques and commands to get more powerful analysis**"},{"metadata":{},"id":"bdefd561","cell_type":"markdown","source":"## **FUNCTIONS, VARS, INIT**"},{"metadata":{},"id":"ad8248a7","cell_type":"markdown","source":"**We create a couple of variables we will use later: home_path (home path of the created environment) and thisnb (name of the current notebook).**"},{"metadata":{"trusted":true},"id":"5d2c0211","cell_type":"code","source":"home_path=!pwd\nthisnb=home_path[0]+\"/ah2polivio-fls.ipynb\"","execution_count":1,"outputs":[]},{"metadata":{},"id":"87937116","cell_type":"markdown","source":"## **IMPORTS & INIT**"},{"metadata":{},"id":"2b0407e5","cell_type":"markdown","source":"**We load all the needed libraries defined in the imports.ipynb notebook and deactivate the warnings in order to have a clean result.**"},{"metadata":{"trusted":true},"id":"ebdce879","cell_type":"code","source":"%run imports.ipynb","execution_count":2,"outputs":[]},{"metadata":{},"id":"382077a9","cell_type":"markdown","source":"## **READ DATA**"},{"metadata":{},"id":"fe4c9128","cell_type":"markdown","source":"**We download the outputs we got from running kape in the ah2polivio evidence.**"},{"metadata":{"trusted":true},"id":"3c250212","cell_type":"code","source":"url = 'https://drive.google.com/uc?id=1nUebvaM2OL_id2_1K8pKXHIpFpt-nsaC'\nof = 'ah2polivio_kape_MFT.tgz'\ngdown.download(url, of, quiet=False)\ngdown.extractall(\"ah2polivio_kape_MFT.tgz\", to=\"./ah2polivio_kape_MFT\")","execution_count":3,"outputs":[{"output_type":"stream","text":"Downloading...\nFrom: https://drive.google.com/uc?id=1nUebvaM2OL_id2_1K8pKXHIpFpt-nsaC\nTo: /home/jovyan/ah2polivio_kape_MFT.tgz\n100%|██████████| 515k/515k [00:00<00:00, 61.0MB/s]\n","name":"stderr"},{"output_type":"execute_result","execution_count":3,"data":{"text/plain":"['./ah2polivio_kape_MFT/FileSystem',\n './ah2polivio_kape_MFT/FileSystem/20201102233826_MFTECmd_$J_Output.csv',\n './ah2polivio_kape_MFT/FileSystem/20201102233807_MFTECmd_$Boot_Output.csv',\n './ah2polivio_kape_MFT/FileSystem/20201102233820_MFTECmd_$MFT_Output.csv',\n './ah2polivio_kape_MFT/FileSystem/20201102233831_MFTECmd_$SDS_Output.csv']"},"metadata":{}}]},{"metadata":{},"id":"34a337e9","cell_type":"markdown","source":"**Let's read the kape outputs with the xread() function from ds4n6_lib. It will generate the .pkl file we will use with the dataframes and store the collection of dataframes in d4.out**"},{"metadata":{"trusted":true},"id":"dfd2070a","cell_type":"code","source":"%%time\n\nd4.debug = 0\n\n# Read raw data\nxread(tool=\"kape\", rootpath=home_path[0]+\"/ah2polivio_kape_MFT\", folder_parsing_mode=\"single_host_with_categories\", path_prefix=\"G__exercises_kape_output-jg\", pluginisdfname=True, notebook_file=thisnb)","execution_count":4,"outputs":[{"output_type":"stream","text":"- Searching notebook for saved input file / folder (_kaped)\n  + Found: /home/jovyan/ah2polivio_kape_MFT\n\n- Reading file(s):\n  + Reading csv file 20201102233807_MFTECmd_$Boot_Output.csv                                                                        into dataframe ->  FileSystem-MFTECmd_$Boot                                                                             \n  + Reading csv file 20201102233820_MFTECmd_$MFT_Output.csv                                                                         into dataframe ->  FileSystem-MFTECmd_$MFT                                                                              \n  + Reading csv file 20201102233831_MFTECmd_$SDS_Output.csv                                                                         into dataframe ->  FileSystem-MFTECmd_$SDS                                                                              \n  + Reading csv file 20201102233826_MFTECmd_$J_Output.csv                                                                           into dataframe ->  FileSystem-MFTECmd_$J                                                                                \n\n\nNOTE: Now you can use the syntax <yourvar>['Category'] to access your dataframe\n\n- Saving Harmonized DataFrames dict (dfs) as pickle file:\n    /home/jovyan/ah2polivio_kape_MFT.ham.pkl\n- Done.\n\n- Reading Done.\n\n- Recover the object read by executing:  myvar = d4.out (e.g. mydf = d4.out)\n\nCPU times: user 475 ms, sys: 120 ms, total: 595 ms\nWall time: 609 ms\n","name":"stdout"}]},{"metadata":{},"id":"7b4b499f","cell_type":"markdown","source":"**We store the dataframes in a more friendly variable: kpdfs.**"},{"metadata":{"trusted":true},"id":"3f6927b9","cell_type":"code","source":"kpdfs = d4.out","execution_count":10,"outputs":[]},{"metadata":{"trusted":true},"id":"0ffec507","cell_type":"code","source":"# Automatically created - DO NOT EDIT OR REMOVE unless you want to change the file to read (in that case, remove this cell)\n# 2021-04-20 10:32:07.685322\n# _kaped_f2read = \"/home/jovyan/ah2polivio_kape_MFT\"","execution_count":6,"outputs":[]},{"metadata":{},"id":"412c8bb8","cell_type":"markdown","source":"**This is an output generated by the framework that is used to let you know the input you are reading.**"},{"metadata":{},"id":"fe5bc157","cell_type":"markdown","source":"## **ANALYSIS**"},{"metadata":{},"id":"59b61710","cell_type":"markdown","source":"### **GENERIC**"},{"metadata":{},"id":"7dd6b3f4","cell_type":"markdown","source":"**xdisplay is used to configure the settings of the default dataframe that will be displayed in the next cell.**"},{"metadata":{"trusted":true},"id":"3a9000a7","cell_type":"code","source":"xdisplay()","execution_count":11,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Markdown object>","text/markdown":"#### Pandas options"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"IntText(value=60, description='display.max_rows: ', style=DescriptionStyle(description_width='initial'))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"b83cc93acf644480b3cb5d30d5ea5251"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"IntText(value=10, description='display.min_rows: ', style=DescriptionStyle(description_width='initial'))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"674c807a155b4e7483119978d19c13f5"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"IntText(value=20, description='display.max_columns: ', style=DescriptionStyle(description_width='initial'))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"75513480c2e54ac1862aa8d58bb7381e"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Dropdown(description='display.colheader_justify: ', options=('right', 'left'), style=DescriptionStyle(descript…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"a0c5e8a217d14409aa9957d88f1563ef"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Checkbox(value=True, description='display.expand_frame_repr', indent=False, style=DescriptionStyle(description…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"9b23cc896f49441fab28eaa6db1bef8a"}},"metadata":{}}]},{"metadata":{},"id":"290830ef","cell_type":"markdown","source":"**We use the xmenu is to select and display the dataframe we want.** "},{"metadata":{"scrolled":true,"trusted":true},"id":"ccc6fbd3","cell_type":"code","source":"xmenu(kpdfs)","execution_count":12,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Markdown object>","text/markdown":"**DataFrame visualization menu:**"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Box(children=(Dropdown(description='Select DataFrame', layout=Layout(width='550px'), options=('Select DataFram…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e5556989e4fa4a9397eecca31c4f872a"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"HBox(children=(Label(value='Selected dataframe: '),))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"8b561da0174d4fd5910107141527f1ac"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"HBox(children=(HTML(value=\"<div style='color: orange; font-weight: bold;'>Data Loading. Please Wait…</div>\"),)…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"3a5569b70f844e5799a6965db05f364e"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Output(layout=Layout(overflow='auto', width='99%'))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"288e1ba43704458382c29d680d2e98f5"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Output(layout=Layout(overflow='auto', width='99%'))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"24677321db624c9685e97c41fa61bbc7"}},"metadata":{}},{"output_type":"execute_result","execution_count":12,"data":{"text/plain":"<ds4n6_lib.gui.dfs_explorer at 0x7f0868492a10>"},"metadata":{}}]},{"metadata":{},"id":"d50903b9","cell_type":"markdown","source":"**With xanalysis function you can run different predefined analysis. You can use it with a dataframe or a collection (dictionary).**"},{"metadata":{"trusted":true},"id":"4ef0ad09","cell_type":"code","source":"xanalysis(kpdfs)","execution_count":13,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Markdown object>","text/markdown":"**Analisys explorer:**"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Box(children=(Dropdown(description='Analysis object: ', layout=Layout(width='250px'), options=('Select one...'…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"a89c2ad6ee81491d91443e767e503fbb"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Box(children=(Dropdown(description='Available analysis types: ', layout=Layout(width='550px'), options=(), sty…","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"895e76a1b1e04a149adc3aab81997fcd"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Output(layout=Layout(overflow='auto', width='99%'))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"ff9a1b2e383b47de87a3d5228e13e607"}},"metadata":{}}]},{"metadata":{},"id":"e2248a60","cell_type":"markdown","source":"**This is just an introduction to the new ds4n6_lib, showing some of the analysis we can perform with just a few instructions. During the next weeks we will be publishing different blog posts explaining new techniques and commands to get more powerful analysis**"}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.7.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":5}