Replace String within Multiple Files

[sam-mininberg]

Hi! I started using script recently, which I've found very useful. One pattern I've found myself using is making in-place replacements across multiple files like this:

script.ListFiles("*.txt").ExecForEach("sed -i 's/tpo/typo/g' {{.}}").Wait()

I think it would be nice to have a "ReplaceInEach" function so that I could replace all sed calls with script equivalents. For example:

script.ListFiles("*.txt").ReplaceInEach("tpo", "typo")

Please let me know what you think. Thanks!

[bitfield]

Thanks @sam-mininberg, great suggestion!

I can see where this would be useful, but I'm not sure how well it fits into the existing model of "everything is a pipe" that script uses. There's no output from ReplaceInEach that could be used by subsequent pipeline stages, as far as I can work out.

This might be better implemented as a custom Filter function, I think.

[sam-mininberg]

Thanks for your quick reply, @bitfield!

You make a good point about the output from ReplaceInEach. Perhaps it could return the number of replacements made in each file, in the same format as Freq? That way one could verify that the expected replacements were made. A subsequent Filter could then be applied, for example, to get the list of files in which at least 1 replacement was made.

No worries if ReplaceInEach doesn't make sense for script right now. I like your idea of implementing this with Filter! I'll work on that and follow up here.

[sam-mininberg]

Here's what I came up with:

// ReplaceInEach reads paths from the pipe, one per line, and replaces within
// each file all occurences of the string search with the string replace.
// Returns the paths prefixed with the number of replacements made in each.
func (p *Pipe) ReplaceInEach(search, replace string) *Pipe {
	return p.Filter(func(r io.Reader, w io.Writer) error {
		scanner := newScanner(r)
		for scanner.Scan() {
			file := scanner.Text()
			s, err := File(file).String()
			if err != nil {
				return err
			}
			count := strings.Count(s, search)
			_, err = Echo(strings.ReplaceAll(s, search, replace)).WriteFile(file)
			if err != nil {
				return err
			}
			_, err = fmt.Fprintf(w, "%d %s\n", count, file)
			if err != nil {
				return err
			}
		}
		return scanner.Err()
	})
}

Please let me know what you think, thanks!

[bitfield]

Nice! Could you use FilterLine instead to save having to write the scanner loop?

[sam-mininberg]

Thanks for the feedback @bitfield! Sure, here's my updated implementation:

// ReplaceInEach reads paths from the pipe, one per line, and replaces within
// each file all occurences of the string search with the string replace.
// Returns the paths along with the number of replacements made in each.
func (p *Pipe) ReplaceInEach(search, replace string) *Pipe {
	return p.FilterLine(func(file string) string {
		s, err := File(file).String()
		if err != nil {
			return fmt.Sprintf("%s %s", file, err)
		}
		count := strings.Count(s, search)
		_, err = Echo(strings.ReplaceAll(s, search, replace)).WriteFile(file)
		if err != nil {
			return fmt.Sprintf("%s %s", file, err)
		}
		return fmt.Sprintf("%s %d", file, count)
	})
}

Since FilterLine doesn't return an error like Filter does, I put any error that occurs in the output of the Pipe. As such, I swapped the places of the file name and the count/error (I think [file] [error] is easier to parse than [error] [file]).

Please let me know what you think. Thanks!

[bitfield]

Nice! I think this is sufficiently simple and elegant that the library itself doesn't really need a built-in ReplaceInEach function: your implementation is just fine, and people can customise it to their specific needs without being constrained to a "one size fits all" set of choices.

Would you like to produce a PR to add this example to the README?

[sam-mininberg]

That sounds good! I'll start working on that.