1.1 基于Go构建滴滴核心业务平台的实践

# 服务治理

  • 异常定位
    • 日志格式混乱
      • 大量 adaptor
      • 人工配置与分析
      • 处理性能低,资源消耗大
    • 服务串联困难
      • 上下游定位困难
      • 跨业务定位效率低
      • 缺乏服务调用拓扑关系
    • 链路难以分析
      • 日志孤立
      • 性能要素缺失
  • 链路优化
    • 全链路压测
  • 服务迁移
    • 接口一致性兼容/代理/切流(旁路引流/流量切换/线上观察)

  • 日志规范化, 服务日志规范/通信接口规范/日志组件
    • 数据结构化
      • DLTAG
      • Key/Value
    • 请求串联
      • TraceId
    • 链路可视化
      • SpanId
    • 处理统一化, 处理系统/日志统一处理
      • 采集/计算/存储

# GC

  • 对象数量过多,GC 三色算法耗费较多 CPU
    • 减少对象分配?用值类型代替对象类型?
    • 多用栈,少用堆?Goroutine 拷贝栈扩展?

1.2 Go在Grab地理服务中的实践

# Nearby Service

  • Node.js + PostGIS
    • R-Tree
      • 空间数据存储 一种平衡树
      • 高频写 触发频繁再平衡
  • Golang + Geohash + Redis
    • Geohash
      • 一维字符串表示二维坐标数据(wtw6j89guz9y -> [31.292494, 121.533371])
      • 具有相同前缀的 Geohash 字符串地理位置相近
      • 基于 SortedSet 做 Range 查询
    • CPU/IO 非常高,内存非常低
    • 不能横向(水平)扩展
  • Sharding 扩展
    • 司机空间索引 -> Shard -> Node
    • 空间索引
      • 地球平面网格划分 Grid (Shard)
      • 分层Grid (Cell)
      • Shard -> Cell (0,1,2,…N) / Cell -> (LRU Queue)
    • 一致性哈希
      • Ketema algorithm
      • Replica supported
  • 分布式
    • 最终一致性 AP
    • Serf
      • 基于 Gossip 的 Membership
      • 故障检测
    • SWIM
      • Scable Weakly-consistent Infection-style process group Memmbership protocol
        • 可扩展性一致性感染型进程组成员协议
        • Scable
          • 随着结点的增多 故障检测耗时不会大幅度增加
            • 如 heart-beating 机制 消息通信呈指数级
        • Weakly
          • 在某一时刻不同的结点看到不同的系统状态
          • 最终会收敛到相同的状态
        • Infection-style
          • Gossip
          • 每个结点持有部分结点信息 每个结点与其子结点交换信息
          • 最终所有结点通过‘八卦’别人的信息获取全局信息
        • Membership
          • 找到其他网络结点

1.3 Rethinking Errors for Go 2

package main

import (
        "log"
        "os"
)

func main() {
        r, err := os.Open("let.go")
        if err != nil {
                log.Fatalf("oops: %v", err)
        }
        defer r.Close()
}
// Output
// 2018/04/16 14:19:51 oops: open let.go: no such file or directory

# Go 2 Draft

  • handle err { … }
  • defer err { … }
  • try
func writeToGS(c net.Context, bkt, dst string, r io.Reader) (err error) {
        w := client.Bucket(bkt).Object(dst).NewWriter(c)
        err = errPanicking
        defer func() {
                if err != nil {
                        _ = w.CloseWithError(err)
                } else if err = w.Close(); err != nil {
                        err = fmt.Errorf("oops: %v", err)
                }
        }()
        if _, err = io.Copy(w, r); err != nil {
                return fmt.Errorf("oops: %v", err)
        }
        return nil
}

var errPanicking = errors.New("panicking")

func writeToGS(c context, bkt, dst string, r io.Reader) error {
        handle err { return errors.Wrap(err) }
        w := client.Bucket(bkt).Object(dst).NewWriter(c)
        defer err { try w.CloseWithError(err) }
        try io.Copy(w, r)
        return nil
}

1.4 Go在区块链的发展和演进

# 区块链是什么

  • 去中心化系统 (多中心化?)
  • 数字化账本
  • 不可篡改
  • 确定性的可复制状态机

# 区块链的特点

  • 去中心化、弱中心化
  • 弱信任、对等的写入权限的数据库
  • 共识信任机制,信任来自规则,非第三方
  • 不可篡改
  • 加密安全、强规则
  • 可编程
  • 匿名性
  • 跨平台

# 区块链开发的特点

  • 分布式
    • 网络编程
      • 多线程
    • 使用大量的算法和数据结构
    • 强规则
      • 密码学

# Go 的优势

  • 部署简单,直接编译成机器码
  • 语言级并发支持
  • 性能高
  • 良好的 C 语言的支持
  • 自动垃圾回收
  • Go 是工程上设计良好的语言,比如代码风格一致,自带工具链
  • 代码简洁
  • 静态类型
  • 丰富的标准库
  • 跨平台编译

# Go 思维

  • 全面简单
    • 简洁
    • 类型推断
    • GC
    • 25个 keyword
    • package
  • 正交组合
    • interface 与其实现之间无显示关联
    • 通过组合架构让程序静态结构
    • 垂直组合 (类型组合, type embedding)
    • 水平组合, 通过 interface 进行组合
  • 偏好并发
    • goroutines
    • select
    • channels
      • 模块之间解耦
      • 并发锁
      • 消息通信

# 共识协议

  • POW / POS / DPOS / DBFT / DAG / …

1.5 Badger_ Fast Key-Value DB in Go

  • Cgo is not Go
  • RocksDB / BoltDB
  • LSM trees / B+ trees

1.6 Golang在阿里巴巴调度系统Sigma中的实践

package main

import "fmt"

func main() {
        m := map[int][]byte{
                0xc4200761c0: []byte{228, 189, 160, 229, 165, 189, 229, 147, 135},
                0xc4200761c8: []byte{230, 189, 152, 233, 147, 129, 230, 159, 177},
        }

        for _, v := range m {
                fmt.Printf("%p -> %s\n", &v, v)
        }
}

// Output 1
// 0xc420090020 -> 潘铁柱
// 0xc420090020 -> 你好哇
// 
// Output 2
// 0xc420092020 -> 你好哇
// 0xc420092020 -> 潘铁柱

1.7 罗辑思维Go语言微服务改造实践

1.8 Golang打造下一代互联网-IPFS全解析

2.1 Composition In Go

// Reader is the interface that wraps the basic Read method.
type Reader interface {
        Read(p []byte) (n int, err error)
}

// Writer is the interface that wraps the basic Write method.
type Writer interface {
        Write(p []byte) (n int, err error)
}

// ReadWriter is the interface that groups the basic Read and Write methods.
type ReadWriter interface {
        Reader
        Writer
}

2.3 Bazel build Go

// https://bazel.build

// vgo · golang/go Wiki

// golang/dep: Go dependency management tool

2.4 基于Go-Ethereum构建DPOS机制下的区块链

// Ethereum Project

# 以太坊技术协议,以太坊客户端

  • Geth (Go)
  • Parity (Rust)
  • cpp-ethereum (c++)
  • ethereumj (java)

# 以太坊核心组件

  • Solidity, a new language for smart contracts, 智能合约
  • Web3.js

# 以太坊相关工具组

  • Truffle
  • Metamask
  • mist

# 以太坊外部存储

  • IPFS (Go)
  • Swarm (Go)

# 共识机制对比

  • POW
    • 消耗计算力
    • 出块速度慢,确认慢
    • TPS 极低,10~20
    • 确认1分支+
  • DPOS
    • 代理人模式
    • 出块速度快,确认块
    • TPS 700~1000
    • 平均确认1~3秒

2.5 深入CGO编程

// cgo is not Go | Dave Cheney

package main

/*
#include <stdio.h>

static void _(const char* s) {
    printf("%s\n", s);
}
*/
import "C"

func main() {
    bytes := *new([]byte)
    bytes = append(bytes, 72)
    bytes = append(bytes, 101, 121)
    bytes = append(bytes, 32, 230, 159)
    bytes = append(bytes, 177, 33, 33, 33)
    C._(C.CString(string(bytes)))
}
$ go build -x hello.go
WORK=/tmp/go-build276115419
mkdir -p $WORK/b001/
cd /tmp
CGO_LDFLAGS='"-g" "-O2"' /usr/local/go/pkg/tool/linux_amd64/cgo -objdir ./go-build276115419/b001/ -importpath command-line-arguments -- -I ./go-build276115419/b001/ -g -O2 ./hello.go
cd $WORK
gcc -fno-caret-diagnostics -c -x c - || true
gcc -Qunused-arguments -c -x c - || true
gcc -fdebug-prefix-map=a=b -c -x c - || true
gcc -gno-record-gcc-switches -c -x c - || true
cd $WORK/b001
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x001.o -c _cgo_export.c
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x002.o -c hello.cgo2.c
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_main.o -c _cgo_main.c
cd /tmp
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -o ./go-build276115419/b001/_cgo_.o ./go-build276115419/b001/_cgo_main.o ./go-build276115419/b001/_x001.o ./go-build276115419/b001/_x002.o -g -O2
/usr/local/go/pkg/tool/linux_amd64/cgo -dynpackage main -dynimport ./go-build276115419/b001/_cgo_.o -dynout ./go-build276115419/b001/_cgo_import.go
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile runtime/cgo=/usr/local/go/pkg/linux_amd64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
EOF
/usr/local/go/pkg/tool/linux_amd64/compile -o ./go-build276115419/b001/_pkg_.a -trimpath ./go-build276115419/b001 -p main -buildid mh6f74THdmFYkLXlaEeI/mh6f74THdmFYkLXlaEeI -goversion go1.10.1 -D _/tmp -importcfg ./go-build276115419/b001/importcfg -pack -c=4 ./go-build276115419/b001/_cgo_gotypes.go ./go-build276115419/b001/hello.cgo1.go ./go-build276115419/b001/_cgo_import.go
/usr/local/go/pkg/tool/linux_amd64/pack r ./go-build276115419/b001/_pkg_.a ./go-build276115419/b001/_x001.o ./go-build276115419/b001/_x002.o # internal
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/_pkg_.a # internal
cp $WORK/b001/_pkg_.a /home/x/.cache/go-build/5f/5fbfbbfb3d3561c7d7f9b6eb2595c5b5c662c609289b0928d54557b768cf6c31-d # internal
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile command-line-arguments=$WORK/b001/_pkg_.a
packagefile runtime/cgo=/usr/local/go/pkg/linux_amd64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
packagefile internal/race=/usr/local/go/pkg/linux_amd64/internal/race.a
packagefile sync=/usr/local/go/pkg/linux_amd64/sync.a
packagefile runtime/internal/atomic=/usr/local/go/pkg/linux_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/go/pkg/linux_amd64/runtime/internal/sys.a
packagefile sync/atomic=/usr/local/go/pkg/linux_amd64/sync/atomic.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=IDPnrZESvB0RYrGsJZll/mh6f74THdmFYkLXlaEeI/_kgrX-BOL__ZqaG2o79P/IDPnrZESvB0RYrGsJZll -extld=gcc $WORK/b001/_pkg_.a
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out hello
rm -r $WORK/b001/

2.6 Go与虚拟化容器 runV/Kata

// hyperhq/runv: Hypervisor-based Runtime for OCI

# Secure as VM, Fast as Container

package main

import (
        "fmt"
)

func main() {
        arr := []struct{ v int }{{1}, {v: 2}, {3}}
        for _, e := range arr {
                e.v++
        }
        fmt.Printf("%v\n", arr)

        for idx := range arr {
                arr[idx].v++
        }
        fmt.Printf("%v\n", arr)
}

// Output
// [{1} {2} {3}]
// [{2} {3} {4}]
package main

import (
        "fmt"
)

func main() {
        s0 := []int{1, 2, 3, 4}
        s1 := s0[:2]              // len=2, cap=4, [1, 2]
        s2 := append(s1, 5)       // len=3, cap=4, [1, 2, 5]
        s3 := append(s1, 6, 7)    // len=4, cap=4, [1, 2, 6, 7]
        s4 := append(s1, 8, 9, 0) // len=5, cap=8, [1, 2, 8, 9, 0]
        fmt.Println(s0)
        fmt.Println(s1)
        fmt.Println(s2)
        fmt.Println(s3)
        fmt.Println(s4)
}

// Output
// [1 2 6 7]
// [1 2]
// [1 2 6]
// [1 2 6 7]
// [1 2 8 9 0]

2.7 Go toolchain internals and implementation based on arm64

// How does the go build command work ? | Dave Cheney

# Go toolchain overview

A toolchain is a package composed of the compiler and ancillary tools, libraries and runtime for a language which together allow you to build and run code written in that language.

  • gc: evolved from the Plan 9 toolchain and includes its own compiler, assembler, linker and tools, as well as the Go runtime and standard library.
  • gccgo: extends the gcc project to support Go.
  • llgo: built on top of the LLVM compiler infrastructure.

# Go toolchain example

package main

import "fmt"

func main() {
        bytes := *new([]byte)
        bytes = append(bytes, 72)
        bytes = append(bytes, 101, 121)
        bytes = append(bytes, 32, 230, 159)
        bytes = append(bytes, 177, 33, 33, 33)
        fmt.Printf("%s\n", bytes)
}
$ go build -x hello.go
WORK=/tmp/go-build173481643
mkdir -p $WORK/b001/
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile fmt=/usr/local/go/pkg/linux_amd64/fmt.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
EOF
cd /tmp
/usr/local/go/pkg/tool/linux_amd64/compile -o ./go-build173481643/b001/_pkg_.a -trimpath ./go-build173481643/b001 -p main -complete -buildid 8sirXYvMbbfw92k19xYW/8sirXYvMbbfw92k19xYW -goversion go1.10.1 -D _/tmp -importcfg ./go-build173481643/b001/importcfg -pack -c=4 ./hello.go
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/_pkg_.a # internal
cp $WORK/b001/_pkg_.a /home/x/.cache/go-build/30/30c2c39b4791f0cd3ff6f312df7ef17888d3eb206e0c11fac21287cff9e0cf3b-d # internal
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile command-line-arguments=$WORK/b001/_pkg_.a
packagefile fmt=/usr/local/go/pkg/linux_amd64/fmt.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
packagefile errors=/usr/local/go/pkg/linux_amd64/errors.a
packagefile io=/usr/local/go/pkg/linux_amd64/io.a
packagefile math=/usr/local/go/pkg/linux_amd64/math.a
packagefile os=/usr/local/go/pkg/linux_amd64/os.a
packagefile reflect=/usr/local/go/pkg/linux_amd64/reflect.a
packagefile strconv=/usr/local/go/pkg/linux_amd64/strconv.a
packagefile sync=/usr/local/go/pkg/linux_amd64/sync.a
packagefile unicode/utf8=/usr/local/go/pkg/linux_amd64/unicode/utf8.a
packagefile runtime/internal/atomic=/usr/local/go/pkg/linux_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/go/pkg/linux_amd64/runtime/internal/sys.a
packagefile sync/atomic=/usr/local/go/pkg/linux_amd64/sync/atomic.a
packagefile internal/cpu=/usr/local/go/pkg/linux_amd64/internal/cpu.a
packagefile internal/poll=/usr/local/go/pkg/linux_amd64/internal/poll.a
packagefile internal/testlog=/usr/local/go/pkg/linux_amd64/internal/testlog.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile time=/usr/local/go/pkg/linux_amd64/time.a
packagefile unicode=/usr/local/go/pkg/linux_amd64/unicode.a
packagefile internal/race=/usr/local/go/pkg/linux_amd64/internal/race.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=Vbu61g9AuYzF2ufqINOR/8sirXYvMbbfw92k19xYW/iWFW984BSmpZp9MnLdjK/Vbu61g9AuYzF2ufqINOR -extld=gcc $WORK/b001/_pkg_.a
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out hello
rm -r $WORK/b001/

package main

/*
 #include <stdio.h>

 static void _(const char* s) {
     printf("%s\n", s);
 }
*/
import "C"

func main() {
        bytes := *new([]byte)
        bytes = append(bytes, 72)
        bytes = append(bytes, 101, 121)
        bytes = append(bytes, 32, 230, 159)
        bytes = append(bytes, 177, 33, 33, 33)
        C._(C.CString(string(bytes)))
}
$ go build -x hello.go
WORK=/tmp/go-build142064459
mkdir -p $WORK/b001/
cd /tmp
CGO_LDFLAGS='"-g" "-O2"' /usr/local/go/pkg/tool/linux_amd64/cgo -objdir ./go-build142064459/b001/ -importpath command-line-arguments -- -I ./go-build142064459/b001/ -g -O2 ./hello.go
cd $WORK
gcc -fno-caret-diagnostics -c -x c - || true
gcc -Qunused-arguments -c -x c - || true
gcc -fdebug-prefix-map=a=b -c -x c - || true
gcc -gno-record-gcc-switches -c -x c - || true
cd $WORK/b001
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x001.o -c _cgo_export.c
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x002.o -c hello.cgo2.c
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_main.o -c _cgo_main.c
cd /tmp
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -o ./go-build142064459/b001/_cgo_.o ./go-build142064459/b001/_cgo_main.o ./go-build142064459/b001/_x001.o ./go-build142064459/b001/_x002.o -g -O2
/usr/local/go/pkg/tool/linux_amd64/cgo -dynpackage main -dynimport ./go-build142064459/b001/_cgo_.o -dynout ./go-build142064459/b001/_cgo_import.go
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile runtime/cgo=/usr/local/go/pkg/linux_amd64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
EOF
/usr/local/go/pkg/tool/linux_amd64/compile -o ./go-build142064459/b001/_pkg_.a -trimpath ./go-build142064459/b001 -p main -buildid WZbBGQxOraNbiT0WofcS/WZbBGQxOraNbiT0WofcS -goversion go1.10.1 -D _/tmp -importcfg ./go-build142064459/b001/importcfg -pack -c=4 ./go-build142064459/b001/_cgo_gotypes.go ./go-build142064459/b001/hello.cgo1.go ./go-build142064459/b001/_cgo_import.go
/usr/local/go/pkg/tool/linux_amd64/pack r ./go-build142064459/b001/_pkg_.a ./go-build142064459/b001/_x001.o ./go-build142064459/b001/_x002.o # internal
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/_pkg_.a # internal
cp $WORK/b001/_pkg_.a /home/x/.cache/go-build/5b/5bedc3923cfc8c06c00e1d8446bf5dd68c35f112d221a1589a8e4ec2fa4b9b11-d # internal
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile command-line-arguments=$WORK/b001/_pkg_.a
packagefile runtime/cgo=/usr/local/go/pkg/linux_amd64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
packagefile internal/race=/usr/local/go/pkg/linux_amd64/internal/race.a
packagefile sync=/usr/local/go/pkg/linux_amd64/sync.a
packagefile runtime/internal/atomic=/usr/local/go/pkg/linux_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/go/pkg/linux_amd64/runtime/internal/sys.a
packagefile sync/atomic=/usr/local/go/pkg/linux_amd64/sync/atomic.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=4aUladWsuR7j-eqT9vPE/WZbBGQxOraNbiT0WofcS/8tvmjI0-hde6HLKu2J8F/4aUladWsuR7j-eqT9vPE -extld=gcc $WORK/b001/_pkg_.a
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out hello
rm -r $WORK/b001/

# Go toolchian overflow

$ go tool
addr2line
asm
buildid
cgo
compile
cover
dist
doc
fix
link
nm
objdump
pack
pprof
test2json
tour
trace
vet
  • *.go – compile –> $WORK/b001/pkg.a
  • *.s – asm –> *.o – pack –> runtime=$WORK/b002/pkg.a
  • internal/bytealg=$WORK/b003/pkg.a
  • . . .

  • *.a – link –> hello

2.8 Go在探探后端的工程实践