refresh = True should be the default for GCSFS.ls

[rickmcgeer]

I was working with this system today, and noticed that ls wasn't returning all the files in a bucket. After I changed the call from:

 gcs_file_system.ls(my_bucket)

To

 gcs_file_system.ls(my_bucket, refresh=True)

I can appreciate the value of caching as an option, but returning an outdated file listing doesn't match the semantics of a filesystem ls, leading to user confusion.

[martindurant]

I respectfully disagree. The cost of making remote listing calls is significant, and ignoring the cache in general would result in significant performance problems for most users. Note:

• the cache will be cleared in the correct locations following write operations, so you do not need to request a refresh unless you anticipate writes from another source during your session
• the argument use_listings_cache can be used to turn off the cache if you wish, giving you full control for your application.

I recognise that setting the best defaults for the most people is a hard problem.

[martindurant]

(you may also find the listings_expiry_time argument useful)

[rickmcgeer]

Martin, I certainly understand and respect your position. GCSFS is a terrific package and there are tradeoffs to each of these decisions. As a user, my expectation was that ls would behave like the Posix utility; may I suggest a one- or two-sentence comment in the API docs noting that it doesn't by default, and that using refresh = True mimics the Posix behavior?

…

On Mon, Nov 11, 2024 at 7:31 AM Martin Durant ***@***.***> wrote: (you may also find the listings_expiry_time argument useful) — Reply to this email directly, view it on GitHub <#647 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFAHA7B53NXBHAUMF3JKX32ADEVBAVCNFSM6AAAAABRPOEYCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRYGQ2TAMZUGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Rick McGeer ***@***.*** +15103346004

[martindurant]

Thanks, I appreciate the feedback. In fact, having just faced this question, you might be in the best place to provide this extra text. It might go in the GCSFileSystem docstring, in the gcsfs prose documentation, or over at fsspec (since the argument is shared by several filesystems).

[rickmcgeer]

Martin, of course. I'll do this on a branch today and do a PR.

…

-- Rick

On Tue, Nov 12, 2024 at 6:12 AM Martin Durant ***@***.***> wrote: Thanks, I appreciate the feedback. In fact, having just faced this question, you might be in the best place to provide this extra text. It might go in the GCSFileSystem docstring, in the gcsfs prose documentation, or over at fsspec (since the argument is shared by several filesystems). — Reply to this email directly, view it on GitHub <#647 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFAHA4FT5AIWDBKKDGDAML2AIEGLAVCNFSM6AAAAABRPOEYCOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZQGY2DINRVGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Rick McGeer ***@***.*** +15103346004

[rickmcgeer]

I've forked the repo, made the changes, and filed a pull request.