go - Functions look very similar. How to avoid code duplication? - Stack Overflow

admin管理员组

文章数量:1404565

I have a few convenience funcs and types in my utils package I use in several projects. They help me getting objects, array of objects from external apis via urls or custom requests (for auth api parts). Here how they look:

var myClient = &http.Client{Timeout: 10 * time.Second}

func GetJsonAsMap(url string) (hashmap Map, err error) {
    err = GetJson(url, &hashmap)
    return
}

func GetJsonAsMapArray(url string) (mapArray []Map, err error) {
    err = GetJson(url, &mapArray)
    return
}

// Low-level func. Target must be a pointer.
func GetJson(url string, target any) error {
    req, err := http.NewRequest("GET", url, nil)
    // commented out error handling

    resp, err := myClient.Do(req)
    // commented out error handling

    defer resp.Body.Close()

    err = json.NewDecoder(resp.Body).Decode(target)
    // commented out error handling

    return nil
}

It worked great! Lower level func could also be used when I need to populate a well-defined struct. But most of the time I would just use map or maparray convenience funcs because I want to keep pristine copies of objects from api reply, where new fields could be added to objects by api and I don't want to miss them if I don't update associated struct.

This was good times, when everything just worked.

Later I needed to do a few requests simultaneously, as there is no need to send them one after another, this would drastically shorten script execution. And this is what I tried. I added two new multiple versions of the GetJsonAsMap and GetJsonAsMapArray functions:

func GetJsonAsMapMultiple(urls []string) (hashmaps []Map, errs []error) {
    l := len(urls)
    hashmaps = make([]Map, l)
    errs = make([]error, l)
    ch := make(chan int, l)

    for i, url := range urls {
        go func() {
            hashmaps[i], errs[i] = GetJsonAsMap(url)
            ch <- i
        }()
    }

    for range l {
        <-ch
    }

    return
}

func GetJsonAsMapArrayMultiple(urls []string) (mapArrays [][]Map, errs []error) {
    l := len(urls)
    mapArrays = make([][]Map, l)
    errs = make([]error, l)
    ch := make(chan int, l)

    for i, url := range urls {
        go func() {
            mapArrays[i], errs[i] = GetJsonAsMapArray(url)
            ch <- i
        }()
    }

    for range l {
        <-ch
    }

    return
}

Here you immediately see a problem. These are two identical funcs that just differ in types and in underlying helper func they call. I am also limited in that I don't have lower level multiple version of the func (and can't use well-defined structs to populate in multiple fashion).

I think I might have just one low level multiple func returning some generic type, but I'm not sure how to go about that. Maybe there is a better approach. Right now these multiple funcs are plain ugly.

I have a few convenience funcs and types in my utils package I use in several projects. They help me getting objects, array of objects from external apis via urls or custom requests (for auth api parts). Here how they look:

var myClient = &http.Client{Timeout: 10 * time.Second}

func GetJsonAsMap(url string) (hashmap Map, err error) {
    err = GetJson(url, &hashmap)
    return
}

func GetJsonAsMapArray(url string) (mapArray []Map, err error) {
    err = GetJson(url, &mapArray)
    return
}

// Low-level func. Target must be a pointer.
func GetJson(url string, target any) error {
    req, err := http.NewRequest("GET", url, nil)
    // commented out error handling

    resp, err := myClient.Do(req)
    // commented out error handling

    defer resp.Body.Close()

    err = json.NewDecoder(resp.Body).Decode(target)
    // commented out error handling

    return nil
}

It worked great! Lower level func could also be used when I need to populate a well-defined struct. But most of the time I would just use map or maparray convenience funcs because I want to keep pristine copies of objects from api reply, where new fields could be added to objects by api and I don't want to miss them if I don't update associated struct.

This was good times, when everything just worked.

Later I needed to do a few requests simultaneously, as there is no need to send them one after another, this would drastically shorten script execution. And this is what I tried. I added two new multiple versions of the GetJsonAsMap and GetJsonAsMapArray functions:

func GetJsonAsMapMultiple(urls []string) (hashmaps []Map, errs []error) {
    l := len(urls)
    hashmaps = make([]Map, l)
    errs = make([]error, l)
    ch := make(chan int, l)

    for i, url := range urls {
        go func() {
            hashmaps[i], errs[i] = GetJsonAsMap(url)
            ch <- i
        }()
    }

    for range l {
        <-ch
    }

    return
}

func GetJsonAsMapArrayMultiple(urls []string) (mapArrays [][]Map, errs []error) {
    l := len(urls)
    mapArrays = make([][]Map, l)
    errs = make([]error, l)
    ch := make(chan int, l)

    for i, url := range urls {
        go func() {
            mapArrays[i], errs[i] = GetJsonAsMapArray(url)
            ch <- i
        }()
    }

    for range l {
        <-ch
    }

    return
}

Here you immediately see a problem. These are two identical funcs that just differ in types and in underlying helper func they call. I am also limited in that I don't have lower level multiple version of the func (and can't use well-defined structs to populate in multiple fashion).

I think I might have just one low level multiple func returning some generic type, but I'm not sure how to go about that. Maybe there is a better approach. Right now these multiple funcs are plain ugly.

• go
• goroutine

Share Improve this question Follow edited Mar 10 at 18:05 vipaware asked Mar 10 at 14:45 vipawarevipaware 333 bronze badges 0

Add a comment  | 

3 Answers 3

Sorted by: Reset to default 1

Use type parameters to eliminate code duplication:

// GetJson decodes the resource at url to T and returns the result.
func GetJson[T any](url string) (T, error) {
    req, err := http.NewRequest("GET", url, nil)
    // commented out error handling

    resp, err := myClient.Do(req)
    // commented out error handling

    defer resp.Body.Close()

    var target T
    err = json.NewDecoder(resp.Body).Decode(target)
    // commented out error handling

    return target, err
}

// GetJsons decodes each resource at urls to a T and returns
// a slice of the results.
func GetJsons[T any](urls []string) ([]T, []error) {
    errors := make([]error, len(urls))
    targets := make([]T, len(urls))
    var wg sync.WaitGroup
    wg.Add(len(urls))
    for i, url := range urls {
        go func() {
            defer wg.Done()
            targets[i], errors[i] = GetJson[T](url)
        }()
    }
    wg.Wait()
    return targets, errors
}

Example use:

hashmaps, errors := GetJsons[Map](urls)

Reduce some of the code duplication by using standard library types designed for the scenario. Use sync.WaitGroup for your scenario.

func GetJsonAsMapMultiple(urls []string) (hashmaps []Map, errs []error) {
    hashmaps = make([]Map, len(urls))
    errs = make([]error, len(urls))
    var wg sync.WaitGroup
    wg.Add(len(urls))
    for i, url := range urls {
        go func() {
            defer wg.Done()
            hashmaps[i], errs[i] = GetJsonAsMap(url)
        }()
    }
    wg.Wait()
    return
}

Use the reflect package to reduce this type of duplication.

// GetJsons decodes the JSON responses to the slice pointed
// to by target. The target argument must be a pointer to 
// a slice of cheese.
func GetJsons(urls []string, target any) []error {
    errors := make([]error, len(urls))
    v := reflect.ValueOf(target).Elem()
    v.Set(reflect.MakeSlice(v.Type(), len(urls), len(urls)))
    var wg sync.WaitGroup
    wg.Add(len(urls))
    for i, url := range urls {
        go func() {
            defer wg.Done()
            errors[i] = GetJson(url, v.Index(i).Addr().Interface())
        }()
    }
        wg.Wait()
    return errors
}

Replace calls to GetJsonAs*Multiple with calls to GetJsons:

var hashmaps []Map
errors := GetJsons(urls, &hashmaps)

本文标签： goFunctions look very similar How to avoid code duplicationStack Overflow