Q: unflated list of objects while using to_dict('records') I want a list of commit objects but I am getting nested list of commit objects when there is more than two index(two different date) like the following
[[{'sha': 'af10fa07', 'author': 'author1', 'timelessdate': datetime.date(2020, 12, 28), 'insertion': 24.0, 'deletion': 10.0}, {'sha': 'af10fa07', 'author': 'author1', 'timelessdate': datetime.date(2020, 12, 28), 'insertion': 13.0, 'deletion': 0.0}, [{ 'sha': 'h9a0ga80', 'author': 'author1', 'timelessdate': datetime.date(2020, 12, 29), 'insertion': 14.0, 'deletion': 10.0 }]]]

while I want in such way
[{'sha': 'af10fa07', 'author': 'author1', 'timelessdate': datetime.date(2020, 12, 28), 'insertion': 24.0, 'deletion': 10.0}, {'sha': 'af10fa07', 'author': 'author1', 'timelessdate': datetime.date(2020, 12, 28), 'insertion': 13.0, 'deletion': 0.0}, { 'sha': 'h9a0ga80', 'author': 'author1', 'timelessdate': datetime.date(2020, 12, 29), 'insertion': 14.0, 'deletion': 10.0 }]

this is how I have done
def commits_metrics(group):
    shas = []
    for index in list(group.index):
        author = index[0]
        date = pd.to_datetime(index[1]).date()
        filtered_data = filter_df[(filter_df['timelessdate'] == date) & (filter_df['author'] == author)]
        result = filtered_data[['sha', 'author', 'timelessdate', 'insertion', 'deletion']].to_dict('records')
        shas.append(result)
    return shas


after = pd.to_datetime("2020-12-28", utc=True)
before = pd.to_datetime("2021-01-10", utc=True)
filter_df = self.df[(self.df["date"] > after) & (self.df["date"] < before)]
filter_df.insert(loc=3, column='timelessdate', value=pd.to_datetime(self.df['date']).dt.date)
if filter_df.size:
    timed_commits = filter_df.set_index(["date"])
    grouped = timed_commits.groupby(by=["author"])
    resampled = grouped.resample("D").agg(
        {
            "sha": "size",
            "insertion": "sum",
            "deletion": "sum",
            "filepath": "size",
            "merges": "max",
        }
    )
    
    resampled_copy = resampled.loc[:]
    resampled_copy['merges'] = resampled_copy['merges'].fillna(0)
    less_than_zero = resampled_copy.copy()
    resampled_copy['commitType'] = less_than_zero['merges'].apply(lambda merge_col: 'CODE_COMMIT' if  merge_col <=0 else 'MERGE_COMMIT')


    
    if resampled_copy.size:
        result = [
            {
                "author": key,
                "commitInfo": [g for g in group.to_dict(orient="records")],
                "timestamp": [index[1] for index in list(group.index)],
#                 "commits": [find_sha(filter_df, pd.to_datetime(index[1]).date(), index[0])[0] for index in list(group.index)]
                "commits": commits_metrics(group)
            }
            for key, group in resampled_copy.groupby("author")
        ]
        print('result', result)
    else:
        print("empty resampled")
else:
    print("empty filter_df")

commits_metrics is the function which is responsible for generating above response.
How can I have unflatten list of commit objects even if the date are different?
This is my df
    sha timestamp   date    author  message body    age insertion   deletion    filepath    churn   merges
1   1   cae635054   Sat Jun 26 14:51:23 2021 -0400  2021-06-26 18:51:23+00:00   Andrew Clark    `act`: Resolve to return value of scope function (#21759)   When migrating some internal tests I found it annoying that I couldn't  -24 days +12:21:32.839997                   
2   21  cae635054   Sat Jun 26 14:51:23 2021 -0400  2021-06-26 18:51:23+00:00   Andrew Clark    `act`: Resolve to return value of scope function (#21759)   When migrating some internal tests I found it annoying that I couldn't  -24 days +12:21:32.839997   31.0    0.0 packages/react-reconciler/src/__tests__/ReactIsomorphicAct-test.js  31.0    
3   22  cae635054   Sat Jun 26 14:51:23 2021 -0400  2021-06-26 18:51:23+00:00   Andrew Clark    `act`: Resolve to return value of scope function (#21759)   When migrating some internal tests I found it annoying that I couldn't  -24 days +12:21:32.839997   1.0 1.0 packages/react-test-renderer/src/ReactTestRenderer.js   0.0 
4   23  cae635054   Sat Jun 26 14:51:23 2021 -0400  2021-06-26 18:51:23+00:00   Andrew Clark    `act`: Resolve to return value of scope function (#21759)   When migrating some internal tests I found it annoying that I couldn't  -24 days +12:21:32.839997   24.0    14.0    packages/react/src/ReactAct.js  10.0    
5   25  e2453e200   Fri Jun 25 15:39:46 2021 -0400  2021-06-25 19:39:46+00:00   Andrew Clark    act: Add test for bypassing queueMicrotask (#21743) Test for fix added in #21740    -25 days +13:09:55.839997   50.0    0.0 packages/react-reconciler/src/__tests__/ReactIsomorphicAct-test.js  50.0    
6   27  73ffce1b6   Thu Jun 24 22:42:44 2021 -0400  2021-06-25 02:42:44+00:00   Brian Vaughn    DevTools: Update tests to fix warnings/errors (#21748)  Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18)  -26 days +20:12:53.839997   4.0 5.0 packages/react-devtools-shared/src/__tests__/FastRefreshDevToolsIntegration-test.js -1.0    
7   28  73ffce1b6   Thu Jun 24 22:42:44 2021 -0400  2021-06-25 02:42:44+00:00   Brian Vaughn    DevTools: Update tests to fix warnings/errors (#21748)  Some new ones had slipped in (e.g. deprecated ReactDOM.render message from 18)  -26 days +20:12:53.839997   4.0 4.0 packages/react-devtools-shared/src/__tests__/componentStacks-test.js    0.0 

UPDATE
Actually, I am getting the following structure already, but I want the second one. Though I felt extremely difficult to do that so I created a separate commits object which will have similar format as in details inside commitInfo object. I hope I could make my intention and question clear.
[
      {
        "author": "Author1",
        "commitInfo": [
          { # this one is for timestamp 2021-01-12T00:00:00+00:Z
            "sha": 30,
            "insertion": 572, # this is the overall insertion
            "deletion": 495, # this is the overall deletion
            "filepath": 30, # this is the overall file changes
            "merges": 0
          },
          { # this one for 2021-01-13T00:00:00+00:Z
            "sha": 18,
            "insertion": 337,
            "deletion": 47,
            "filepath": 18,
            "merges": 0
          }
        ],
        "timestamp": [
          "2021-01-12T00:00:00+00:Z",
          "2021-01-13T00:00:00+00:Z"
        ]
      },
      {
        "author": "Author2",
        "commitInfo": [
          {
            "sha": 35,
            "insertion": 601,
            "deletion": 127,
            "filepath": 35,
            "merges": 0
          }
        ],
        "timestamp": [
          "2021-01-12T00:00:00+00:Z"
        ]
      },
    ]




     [
      {
        "author": "Author1",
        "commitInfo": [
          { # this one is for timestamp 2021-01-12T00:00:00+00:Z
            "sha": 30,
            "insertion": 572, # this is the overall insertion
            "deletion": 495, # this is the overall deletion
            "filepath": 30, # this is the overall file changes
            "merges": 0,
            "details": [{ sha: 'commitsha', insertion: 10, deletion: 10, date: ''}, { sha: '', insertion: 100, deletion: 80, date: ''}] # detail of all 30 shas
          },
          { # this one for 2021-01-13T00:00:00+00:Z
            "sha": 18,
            "insertion": 337,
            "deletion": 47,
            "filepath": 18,
            "merges": 0
          }
        ],
        "timestamp": [
          "2021-01-12T00:00:00+00:Z",
          "2021-01-13T00:00:00+00:Z"
        ]
      },
      {
        "author": "Author2",
        "commitInfo": [
          {
            "sha": 35,
            "insertion": 601,
            "deletion": 127,
            "filepath": 35,
            "merges": 0
          }
        ],
        "timestamp": [
          "2021-01-12T00:00:00+00:Z"
        ]
      },
    ]


A: The problem you're running into inside commits_metrics() is that you're adding a list, result, to the list shas using shas.append().
From the docs:





Operation
Result




s.append(x)
appends x to the end of the sequence (same as s[len(s):len(s)] = [x])


s.extend(t) or s += t
extends s with the contents of t (for the most part the same as s[len(s):len(s)] = t)




So with append you're adding [result], which creates nested lists. Use shas += result or shas.extend(result) instead.
As or the bigger picture I can't really work out what you're trying to do, so I won't be able to help further unless you somewhat clarify your question.

For a more concise way of doing what you're doing:
>>> groups = df.groupby(['author', pd.Grouper(key='date', freq='D')])
>>> stats = groups.agg({'sha': 'size', 'insertion': 'sum', 'deletion': 'sum', 'filepath': 'size', 'merges': 'max'})
>>> details = groups[['sha', 'insertion', 'deletion']].apply(lambda s: s.to_dict(orient='records'))
>>> stats.join(details.rename('commitInfo'))
                                        sha  insertion  deletion  filepath  merges                                         commitInfo
author       date                                                                                                                    
Andrew Clark 2021-06-25 00:00:00+00:00    1       50.0       0.0         1    50.0  [{'sha': 'e2453e200', 'insertion': 50.0, 'dele...
             2021-06-26 00:00:00+00:00    4       56.0      15.0         4    31.0  [{'sha': 'cae635054', 'insertion': nan, 'delet...
Brian Vaughn 2021-06-25 00:00:00+00:00    2        8.0       9.0         2     0.0  [{'sha': '73ffce1b6', 'insertion': 4.0, 'delet...

That gives you all the info you want with an (author, date) index and the list of commit details as a list in the commitInfo column. You can then transform that however you want.
