Add Files metadata table

[Gowthami03B]

No description provided.

[sungwy]

Hi @Gowthami03B this looks great. I have one request to add the optional snapshot_id argument to support time traveling between different snapshots Iceberg metadata table.

[sungwy]

Hi @HonahX could we get your help in triggering this workflow to see if the CI succeeds?

[Gowthami03B]

Hello @HonahX

Can I get a re-trigger again (the checks succeeds in my PR to my fork, so I am confident)?
Also requesting re-review here , I addressed the comments, hoping we could close this :)

[HonahX]

LGTM! Thanks for adding this @Gowthami03B.

[Fokko]

Sorry for now following up on this @Gowthami03B Could you rebase so we can get this in? Thanks!

[Fokko]

@Gowthami03B gentle ping, this is the last metadata table, and we would love to include this into the release! 🙌

[Gowthami03B]

@Fokko@kevinjqliu@amogh-jahagirdar Can I get a re-review here please? Want to close this asap for the release timeline :)

[kevinjqliu]

LGTM!

r? @Fokko / @HonahX

[HonahX]

LGTM too. @Gowthami03B Thanks for working on this! Thanks everyone for reviewing. Let's get this last metadata table in!

[DieHertz]

Hi guys, sorry if it's not the right place to ask this question.
Do you know of a viable way to speed up table.inspect.files() for large tables?
Maybe something in mind that I could implement and contribute to upstream.

I haven't profiled yet but I guess the gist of the issue is manifest.fetch_manifest_entry being called synchronously and sequentially in a loop.
Offloading this call to a thread-based executor doesn't help much, probably because of GIL, and a process-based executor is harder to implement because of unpicklable types involved.

As of now pyspark's .files metatable collection can be done considerably quicker than pyiceberg's

[kevinjqliu]

I think there's definitely room for improvement. @DieHertz do you mind opening an issue for this?

[DieHertz]

Will do