Q: Understanding error frequencies by parsing server log in python/pandas for web scale app I am running a web-scale application and want to understand the errors experienced by users. The server log captures application errors. I want to parse the log into a dataframe, understand the top x errors and then plot the top x errors by day (both absolute and relative frequencies).
The first step is to parse the log and load it into a dataframe. The raw log file looks like this:


[2017-06-20T12:57:56.549-04:00] [nexteesadf_server1] [WARNING] [ADF_FACES-60099] [oracle.adfinternal.view.faces.renderkit.rich.RegionRenderer] [tid: [ACTIVE].ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: test@test.com] [ecid: 005KaSb9zIX4ulMpMK4Eye0000xO00zuxf,0:1] [APP: netappApp_application1.ear#66.1254] [DSID: 0000Ln6ErTF4mnMpMKL6ie1PHmGo00045R] [URI: /netapp/faces/pages/submission/Submission.jspx] The region component with id: pt1:pt_r1 has detected a page fragment with multiple root components. Fragments with more than one root component may not display correctly in a region and may have a negative impact on performance. It is recommended that you restructure the page fragment to have a single root component.
[2017-06-20T12:53:43.665-04:00] [nexteesadf_server1] [WARNING] [ADF_FACES-60051] [oracle.adfinternal.view.faces.renderkit.rich.table.BaseColumnRenderer] [tid: [ACTIVE].ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: test@test.com] [ecid: 005KaSO5zOR4ulMpMK4Eye0000xO00zuB9,0:1] [APP: netappApp_application1.ear#66.1254] [DSID: 0000Ln6NNvT4mnMpMKL6ie1PHmGo000495] [URI: /netapp/faces/pages/oversight/Oversight.jspx] Invalid width: -20 specified on column: pt1:r1:1:r1:0:r11:1:r2:0:r5:0:pc1:t1:1:c26.
[2017-06-20T12:53:44.324-04:00] [nexteesadf_server1] [ERROR] [] [com.someco.netapp.sourcefile.helper.SourceFileHelper] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: test@test.com] [ecid: 005KaSO8oaU4QtMpMKc9ye0003da0050lR,0:1] [APP: netappApp_application1.ear#66.1254] [DSID: 0000Ln6IpJM4mnMpMKL6ie1PHmGo00047B] [URI: /netapp/faces/pages/submission/Submission.jspx] SourceFileHelper  validateSourceFile: 1152379
[2017-06-20T12:53:44.329-04:00] [nexteesadf_server1] [ERROR] [ADFC-64010] [oracle.adf.controller.internal.binding.TaskFlowRegionModelViewPort] [tid: [ACTIVE].ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: test@test.com] [ecid: 005KaSO8oaU4QtMpMKc9ye0003da0050lR,0:1] [APP: netappApp_application1.ear#66.1254] [DSID: 0000Ln6IpJM4mnMpMKL6ie1PHmGo00047B] [URI: /netapp/faces/pages/submission/Submission.jspx] ADFc: Page '/com/someco/netapp/submission/ui/fragment/uploadSubmissionFile.jsff' references task flow binding 'guidelineNavTF1' more than once which is not supported.


However, the server log also contains lines like the below (which may mess up the parsing):


[2017-06-20T12:56:41.454-04:00] [nexteesadf_server1] [WARNING] [] [oracle.adfinternal.view.faces.lifecycle.LifecycleImpl] [tid: [ACTIVE].ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: test@test.com] [ecid: 005KaSYgDyu4QtMpMKc9ye0003da0051Le,0:1] [APP: netappApp_application1.ear#66.1254] [DSID: 0000Ln6Ro684mnMpMKL6ie1PHmGo0004B6] [URI: /netapp/faces/pages/oversight/Oversight.jspx] ADF_FACES-60098:Faces lifecycle receives unhandled exceptions in phase RESTORE_VIEW 1[[
oracle.adf.controller.internal.AdfcIllegalStateException: oracle.adf.controller.ControllerException: ADFC-12000: State ID in request is invalid for the current session.
 at oracle.adfinternal.controller.state.ControllerState.synchronizeStatePart2(ControllerState.java:551)
 at oracle.adfinternal.controller.application.SyncNavigationStateListener.afterPhase(SyncNavigationStateListener.java:79)
 at oracle.adfinternal.controller.lifecycle.ADFLifecycleImpl$PagePhaseListenerWrapper.afterPhase(ADFLifecycleImpl.java:538)
 at oracle.adfinternal.controller.lifecycle.LifecycleImpl.internalDispatchAfterEvent(LifecycleImpl.java:120)
 at oracle.adfinternal.controller.lifecycle.LifecycleImpl.dispatchAfterPagePhaseEvent(LifecycleImpl.java:168)
 at oracle.adfinternal.controller.faces.lifecycle.ADFPhaseListener$PhaseInvokerImpl.dispatchAfterPagePhaseEvent(ADFPhaseListener.java:310)
 at oracle.adfinternal.controller.faces.lifecycle.ADFPhaseListener.afterPhase(ADFPhaseListener.java:87)
 at oracle.adfinternal.controller.faces.lifecycle.ADFLifecyclePhaseListener.afterPhasee(ADFLifecyclePhaseListener.java:53)
 at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:520)
 at oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:207)
 at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
 at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
 at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
 at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
 at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:205)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at com.someco.netapp.common.util.SessionHandlingFilter.doFilter(SessionHandlingFilter.java:105)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:128)
 at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
 at oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:60)
 at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:446)
 at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:271)
 at org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:177)
 at org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:92)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at com.someco.netapp.filters.IEVersionModeFilter.doFilter(IEVersionModeFilter.java:50)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at com.someco.netapp.common.util.netappClickjackFilter.doFilter(netappClickjackFilter.java:87)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:180)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.adfinternal.view.faces.caching.filter.AdfFacesCachingFilter.doFilter(AdfFacesCachingFilter.java:137)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:119)
 at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)
 at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:460)
 at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:103)
 at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:171)
 at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.security.wls.filter.SSOSessionSynchronizationFilter.doFilter(SSOSessionSynchronizationFilter.java:418)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730)
 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696)
 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
 at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
 at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273)
 at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179)
 at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: oracle.adf.controller.ControllerException: ADFC-12000: State ID in request is invalid for the current session.
 ... 58 more
]]



I would like to parse the first 4 fields into columns within the dataframe:


  
*
  
*[2017-06-20T12:57:56.549-04:00]
  
*[nexteesadf_server1]
  
*[WARNING]
  
*[ADF_FACES-60099]
  I want to convert the first column into a datetime (numpy) and the next 3 can be strings.
  

Whats the best way to do this?

A: You are gonna have to parse the each line of the log file. Then create a dictionary for each line and use the pandas.Dataframe.from_dict() method to create the dataframe.
import pandas as pd
import numpy as np

def parse_log(filename):
    with open(filename, 'r') as f:
        lines = f.readlines()

    entries = []
    for line in lines:
        entry = {}
        cols = line.split('] [')
        entry['log_date'] = np.datetime64(cols[0].replace('[', ''))
        entry['log_server'] = cols[1]
        entry['log_level'] = cols[2]
        entry['col_four'] = cols[3]  # idk column proper name
        entry['col_five'] = cols[4]  # idk column proper name
        entry['tid'] = cols[5].replace('tid: ', '')
        entry['userId'] = cols[6].replace('userId: ', '')
        entry['ecid'] = cols[7].replace('ecid: ', '')
        entry['app'] = cols[8].replace('APP: ', '')
        entry['dsid'] = cols[9].replace('DSID: ', '')
        # this is where the last column and the message get split
        last = cols[10].split(']')
        entry['uri'] = last[0].replace('URI: ', '')
        entry['message'] = last[1].strip()
        entries.append(entry)
    return entries


if __name__ == '__main__':
    entries = parse_log('example.log')
    df = pd.DataFrame.from_dict(entries)
    print(df.head())

You can now use all the methods available via pandas to analyze the data. I didn't know what the column names for a couple of the columns should be so I just gave them generic names.
