Skip to content

Golang lib for chain functional programming with spark-like APIs.

License

Notifications You must be signed in to change notification settings

Yangruipis/go-functional

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Go-Functional

Features

Go-functional is inspired by PyFunctional and go-funk. It makes golang struct data processing easier, especially when we have a data pipeline. Go-functionl is:

⚡️ FAST
there is no reflect at all
⛓️ CHAIN
chain functional operations
🎬 LAZY
most trainsformation operations are lazy evaluated
✨ SIMPLE
use generic, all you need is a Map , instead of MapInt MapInt32 MapInt64
👥 USER-FRIENDLY
Spark style APIs is provided, maybe LinQ someday.

Installation ( go>=1.18 )

go get github.com/Yangruipis/go-functional

Usage

basic import

 package main

 import (
   "fmt"

   "github.com/Yangruipis/go-functional/pkg/fn"
 )

 func main() {
   fn.RangeSeq(0, 10, 1).ForEach(func(i, v int) {
     fmt.Printf("%d\n", v)
   })
	}

Range -> Map -> Filter -> ForEach

fn.RangeSeq(0, 10, 1).Map(
  func(k, v int) (int, int) {return k, v + 1 },
).Filter(
  func(k, v int) bool { return v >= 3 },
).ForEach(
  func(i, v int) { fmt.Printf("%d\n", v) },
)
3
4
5
6
7
8
9
10

Range -> Map -> GroupByKey -> Aggregate -> ForEach

fn.RangeSeq(0, 10, 1).Map(
  func(k, v int) (int, int) {return k%2, v }, // split to 2 groups
).GroupByKey().Aggregate(func(vv []int) int {
  s := 0
  for _, v := range vv {
    s += v
  }
  return s
}).ForEach(
  func(k int, v int) { fmt.Printf("sum of group %d is: %v\n", k, v) },
)
mean of group 0 is: 20
mean of group 1 is: 25

multi path with cache: Range -> Map -> Cache ( –> Filter ) & ( –> Shuffle )

c := fn.RangeSeq(0, 10, 1).Map(
  func(k, v int) (int, int) { return k, v * 2 },
).Cache()

c1 := c.Filter(func(k, v int) bool { return v >= 10 } )
fmt.Printf("paths: %v, results: %v \n", c1.Paths, c1.ToSlice())

c2 := c.Shuffle()
fmt.Printf("paths: %v, results: %v \n", c2.Paths, c2.ToSlice())
paths: [Map Cache Filter], results: [10 12 14 16 18] 
paths: [Map Cache Shuffle], results: [2 14 8 0 18 4 6 10 16 12]

API list

There are two types of API(consistent with Spark):

  1. Transformation: Iterator in, iterator out. Will not be executed until action operation. Iterator is supposed to be consumed only once.
  2. Action: Iterator in, results out. All transformation operations are executed here (lazy exec).
func call[51/51]chain call[33/51]namesignaturetype
Map[K, V] -> [K, V]transformation
Filter[K, V] -> [K, V]transformation
Flatten[K, []V] -> [K, V]transformation
GroupBy[K, V] -> [K, []V]transformation
GroupByKey[K, V] -> [K, []V]transformation
GroupByVal[K, V] -> [V, []K]transformation
FlatMap[K, []V] -> [K, []V]transformation
ReduceByKey[K, []V] -> [K, V]transformation
CountByKey[K, V] -> [K, int]transformation
CountByVal[K, V] -> [V, int]transformation
Union[K, V] [K, V] -> [K, V]transformation
Intersect[K, V] [K, V] -> [K, V]transformation
Subtract[K, V] [K, V] -> [K, V]transformation
Distinct[K, V] -> [K, V]transformation
UnionBy[K, V] [K, V] -> [K, V]transformation
IntersectBy[K, V] [K, V] -> [K, V]transformation
SubtractBy[K, V] [K, V] -> [K, V]transformation
DistinctBy[K, V] -> [K, V]transformation
Cartesian[K, V] [K, V] -> [K, V]transformation
Chunk[K, V] -> [K, []V]transformation
Sort[K, V] -> [K, V]transformation
Aggregate[K, V] -> [K, V1]transformation
Zip[K, V] [K, V1]-> [K, [V, V1]]transformation
Invert[K, V] -> [V, K]transformation
Reverse[K, V] -> [K, V]transformation
Shuffle[K, V] -> [K, V]transformation
Sample[K, V] -> [K, V]transformation
Choices[K, V] -> [K, V]transformation
Head[K, V] -> [K, V]transformation
Tail[K, V] -> [K, V]transformation
Cache[K, V] -> [K, V]transformation
RepeatV -> [int, V]transformation(init)
Rangeint -> [int, int]transformation(init)
Reduce[K, V] -> Vaction
Size[K, V] -> intaction
Exists[K, V] -> boolaction
ExistsBy[K, V] -> boolaction
All[K, V] -> boolaction
Any[K, V] -> boolaction
Count[K, V] -> intaction
CountBy[K, V] -> intaction
ToSlice[K, V] -> []Vaction
ToMap[K, V] -> map[K]Vaction
ToSet[K, V] -> map[K]struct{}action
Sum[K, V] -> intaction
Avg[K, V] -> intaction
ForEach[K, V] -> voidaction
Entries[K, V] -> [][K, V]action
IndexOf[K, V] -> []intaction
NIndexOf[K, V] -> intaction
Values[K, V] -> []Vaction
Keys[K, V] -> []Kaction

Development

run tests

go mod download

go test ./...

contributing