Mem usage and Increment equal for many lines

[PiotrZiolo]

I was searching for a memory leak in my code and for debugging purposes I split a for loop into separate lines where I fill a dict with some objects. These objects are created there. Here's the report from memory_profiler:

Line # Mem usage Increment Line Contents
================================================
 44 1125.984 MiB 1125.984 MiB @profile
 45 def __init__(self, priors=None, base_classes=None, seed=9):
 46 """
 47 :param pd.DataFrame priors: DataFrame with three columns: date, hour_of_week,
 48 prior. The last column defines priors (in the form of dictionaries)
 49 for given dates and hours of week.
 50 :param pd.DataFrame base_classes: DataFrame with three columns: date, hour_of_week,
 51 base_class. The last column defines priors (in the form of dictionaries)
 52 for given dates and hours of week.
 53 :param int seed: Seed for the random number generator.
 54 """
 55 
 56 1125.984 MiB 0.000 MiB self.models ={} # type: Dict[str, CompetitiveClickProbabilityModule]
 57 
 58 1125.984 MiB 0.000 MiB self.seed = seed
 59 1125.984 MiB 0.000 MiB self.rng = np.random.RandomState(seed)
 60 1125.984 MiB 0.000 MiB self.priors = priors
 61 1125.984 MiB 0.000 MiB self.base_classes = base_classes
 62 
 63 1125.984 MiB 0.000 MiB seed_min = 100000
 64 1125.984 MiB 0.000 MiB seed_max = 999999
 65 1125.984 MiB 0.000 MiB seeds = self.rng.randint(low=seed_min, high=seed_max, size=len(self.priors))
 66 
 67 1126.316 MiB 0.332 MiB base_df = priors.copy()
 68 1126.387 MiB 0.070 MiB base_df.loc[:, "base_class"] = base_classes["base_class"]
 69 1127.254 MiB 0.867 MiB base_df.loc[:, "seed"] = seeds
 70 
 71 1127.254 MiB 0.000 MiB self.models ={} # type: Dict[str, SimulatorModule]
 72 
 73 1127.383 MiB 1127.383 MiB self.models["{}.{}".format(base_df["date"][0], base_df["hour_of_week"][0])] = base_df["base_class"][0](base_df["prior"][0], base_df["seed"][0])
 74 1127.508 MiB 1127.508 MiB self.models["{}.{}".format(base_df["date"][1], base_df["hour_of_week"][1])] = base_df["base_class"][1](base_df["prior"][1], base_df["seed"][1])
 75 1127.898 MiB 1127.898 MiB self.models["{}.{}".format(base_df["date"][2], base_df["hour_of_week"][2])] = base_df["base_class"][2](base_df["prior"][2], base_df["seed"][2])
 76 1128.383 MiB 1128.383 MiB self.models["{}.{}".format(base_df["date"][3], base_df["hour_of_week"][3])] = base_df["base_class"][3](base_df["prior"][3], base_df["seed"][3])
 77 1128.758 MiB 1128.758 MiB self.models["{}.{}".format(base_df["date"][4], base_df["hour_of_week"][4])] = base_df["base_class"][4](base_df["prior"][4], base_df["seed"][4])
 78 1129.176 MiB 1129.176 MiB self.models["{}.{}".format(base_df["date"][5], base_df["hour_of_week"][5])] = base_df["base_class"][5](base_df["prior"][5], base_df["seed"][5])
 79 1129.633 MiB 1129.633 MiB self.models["{}.{}".format(base_df["date"][6], base_df["hour_of_week"][6])] = base_df["base_class"][6](base_df["prior"][6], base_df["seed"][6])
 80 1130.070 MiB 1130.070 MiB self.models["{}.{}".format(base_df["date"][7], base_df["hour_of_week"][7])] = base_df["base_class"][7](base_df["prior"][7], base_df["seed"][7])
 81 1130.508 MiB 1130.508 MiB self.models["{}.{}".format(base_df["date"][8], base_df["hour_of_week"][8])] = base_df["base_class"][8](base_df["prior"][8], base_df["seed"][8])
 82 1130.883 MiB 1130.883 MiB self.models["{}.{}".format(base_df["date"][9], base_df["hour_of_week"][9])] = base_df["base_class"][9](base_df["prior"][9], base_df["seed"][9])
 83 1131.320 MiB 1131.320 MiB self.models["{}.{}".format(base_df["date"][10], base_df["hour_of_week"][10])] = base_df["base_class"][10](base_df["prior"][10], base_df["seed"][10])
 84 1131.758 MiB 1131.758 MiB self.models["{}.{}".format(base_df["date"][11], base_df["hour_of_week"][11])] = base_df["base_class"][11](base_df["prior"][11], base_df["seed"][11])
 85 1132.156 MiB 1132.156 MiB self.models["{}.{}".format(base_df["date"][12], base_df["hour_of_week"][12])] = base_df["base_class"][12](base_df["prior"][12], base_df["seed"][12])
 86 1132.633 MiB 1132.633 MiB self.models["{}.{}".format(base_df["date"][13], base_df["hour_of_week"][13])] = base_df["base_class"][13](base_df["prior"][13], base_df["seed"][13])
 87 1133.008 MiB 1133.008 MiB self.models["{}.{}".format(base_df["date"][14], base_df["hour_of_week"][14])] = base_df["base_class"][14](base_df["prior"][14], base_df["seed"][14])
 88 1133.434 MiB 1133.434 MiB self.models["{}.{}".format(base_df["date"][15], base_df["hour_of_week"][15])] = base_df["base_class"][15](base_df["prior"][15], base_df["seed"][15])
 89 1133.883 MiB 1133.883 MiB self.models["{}.{}".format(base_df["date"][16], base_df["hour_of_week"][16])] = base_df["base_class"][16](base_df["prior"][16], base_df["seed"][16])
 90 1134.320 MiB 1134.320 MiB self.models["{}.{}".format(base_df["date"][17], base_df["hour_of_week"][17])] = base_df["base_class"][17](base_df["prior"][17], base_df["seed"][17])
 91 1134.758 MiB 1134.758 MiB self.models["{}.{}".format(base_df["date"][18], base_df["hour_of_week"][18])] = base_df["base_class"][18](base_df["prior"][18], base_df["seed"][18])
 92 1135.141 MiB 1135.141 MiB self.models["{}.{}".format(base_df["date"][19], base_df["hour_of_week"][19])] = base_df["base_class"][19](base_df["prior"][19], base_df["seed"][19])
 93 1135.570 MiB 1135.570 MiB self.models["{}.{}".format(base_df["date"][20], base_df["hour_of_week"][20])] = base_df["base_class"][20](base_df["prior"][20], base_df["seed"][20])
 94 1136.008 MiB 1136.008 MiB self.models["{}.{}".format(base_df["date"][21], base_df["hour_of_week"][21])] = base_df["base_class"][21](base_df["prior"][21], base_df["seed"][21])
 95 1136.418 MiB 1136.418 MiB self.models["{}.{}".format(base_df["date"][22], base_df["hour_of_week"][22])] = base_df["base_class"][22](base_df["prior"][22], base_df["seed"][22])
 96 1136.883 MiB 1136.883 MiB self.models["{}.{}".format(base_df["date"][23], base_df["hour_of_week"][23])] = base_df["base_class"][23](base_df["prior"][23], base_df["seed"][23])

As you can see starting from line 73 the Increment seems to be broken as it is always equal to Mem usage. The Mem usage suggests that it should be below 1MB. Is it a bug or my misunderstanding how it is supposed to work?

[soundgnome]

I'm seeing the same thing, it appears (in my case) to happen specifically when calling one profiled function from inside another. On lines which call a profiled function, Increment is equal to Mem usage, otherwise Increment is equal to the difference between the last two Mem usage values.

[fabianp]

thanks for reporting this. If you find the bug, please do send a pull request :-)

[eden-wang]

I'm seeing the same thing, it appears (in my case) to happen specifically when calling one profiled function from inside another. On lines which call a profiled function, Increment is equal to Mem usage, otherwise Increment is equal to the difference between the last two Mem usage values.

I met the same problem when one profiled function calls another profiled function inside.

[brianzhang01]

You may want to rebuild and try again now that #280 is in to see if that solves your problem.