GopherChina 2018

1.1 基于Go构建滴滴核心业务平台的实践
1.2 Go在Grab地理服务中的实践
1.3 Rethinking Errors for Go 2
1.4 Go在区块链的发展和演进
1.5 Badger_ Fast Key-Value DB in Go
1.6 Golang在阿里巴巴调度系统Sigma中的实践
1.7 罗辑思维Go语言微服务改造实践
1.8 Golang打造下一代互联网-IPFS全解析
2.1 Composition In Go
2.3 Bazel build Go
2.4 基于Go-Ethereum构建DPOS机制下的区块链
2.5 深入CGO编程
2.6 Go与虚拟化容器 runV/Kata
2.7 Go toolchain internals and implementation based on arm64
2.8 Go在探探后端的工程实践

1.1 基于Go构建滴滴核心业务平台的实践

# 服务治理

异常定位
- 日志格式混乱
  - 大量 adaptor
  - 人工配置与分析
  - 处理性能低，资源消耗大
- 服务串联困难
  - 上下游定位困难
  - 跨业务定位效率低
  - 缺乏服务调用拓扑关系
- 链路难以分析
  - 日志孤立
  - 性能要素缺失
链路优化
- 全链路压测
服务迁移
- 接口一致性兼容/代理/切流(旁路引流/流量切换/线上观察)

日志规范化, 服务日志规范/通信接口规范/日志组件
- 数据结构化
  - DLTAG
  - Key/Value
- 请求串联
  - TraceId
- 链路可视化
  - SpanId
- 处理统一化, 处理系统/日志统一处理
  - 采集/计算/存储

# GC

对象数量过多，GC 三色算法耗费较多 CPU
- 减少对象分配？用值类型代替对象类型？
- 多用栈，少用堆？Goroutine 拷贝栈扩展？

1.2 Go在Grab地理服务中的实践

# Nearby Service

Node.js + PostGIS
- R-Tree
  - 空间数据存储一种平衡树
  - 高频写触发频繁再平衡
Golang + Geohash + Redis
- Geohash
  - 一维字符串表示二维坐标数据(wtw6j89guz9y -> [31.292494, 121.533371])
  - 具有相同前缀的 Geohash 字符串地理位置相近
  - 基于 SortedSet 做 Range 查询
- CPU/IO 非常高，内存非常低
- 不能横向(水平)扩展
Sharding 扩展
- 司机空间索引 -> Shard -> Node
- 空间索引
  - 地球平面网格划分 Grid (Shard)
  - 分层Grid (Cell)
  - Shard -> Cell (0,1,2,…N) / Cell -> (LRU Queue)
- 一致性哈希
  - Ketema algorithm
  - Replica supported
分布式
- 最终一致性 AP
- Serf
  - 基于 Gossip 的 Membership
  - 故障检测
- SWIM
  - Scable Weakly-consistent Infection-style process group Memmbership protocol
    - 可扩展性一致性感染型进程组成员协议
    - Scable
      - 随着结点的增多故障检测耗时不会大幅度增加
        
        如 heart-beating 机制消息通信呈指数级
    - Weakly
      - 在某一时刻不同的结点看到不同的系统状态
      - 最终会收敛到相同的状态
    - Infection-style
      - Gossip
      - 每个结点持有部分结点信息每个结点与其子结点交换信息
      - 最终所有结点通过‘八卦’别人的信息获取全局信息
    - Membership
      - 找到其他网络结点

1.3 Rethinking Errors for Go 2

package main

import (
        "log"
        "os"
)

func main() {
        r, err := os.Open("let.go")
        if err != nil {
                log.Fatalf("oops: %v", err)
        }
        defer r.Close()
}
// Output
// 2018/04/16 14:19:51 oops: open let.go: no such file or directory

# Go 2 Draft

handle err { … }
defer err { … }
try

func writeToGS(c net.Context, bkt, dst string, r io.Reader) (err error) {
        w := client.Bucket(bkt).Object(dst).NewWriter(c)
        err = errPanicking
        defer func() {
                if err != nil {
                        _ = w.CloseWithError(err)
                } else if err = w.Close(); err != nil {
                        err = fmt.Errorf("oops: %v", err)
                }
        }()
        if _, err = io.Copy(w, r); err != nil {
                return fmt.Errorf("oops: %v", err)
        }
        return nil
}

var errPanicking = errors.New("panicking")

func writeToGS(c context, bkt, dst string, r io.Reader) error {
        handle err { return errors.Wrap(err) }
        w := client.Bucket(bkt).Object(dst).NewWriter(c)
        defer err { try w.CloseWithError(err) }
        try io.Copy(w, r)
        return nil
}

1.4 Go在区块链的发展和演进

# 区块链是什么

去中心化系统 (多中心化?)
数字化账本
不可篡改
确定性的可复制状态机

# 区块链的特点

去中心化、弱中心化
弱信任、对等的写入权限的数据库
共识信任机制，信任来自规则，非第三方
不可篡改
加密安全、强规则
可编程
匿名性
跨平台

# 区块链开发的特点

分布式
- 网络编程
  - 多线程
- 使用大量的算法和数据结构
- 强规则
  - 密码学

# Go 的优势

部署简单，直接编译成机器码
语言级并发支持
性能高
良好的 C 语言的支持
自动垃圾回收
Go 是工程上设计良好的语言，比如代码风格一致，自带工具链
代码简洁
静态类型
丰富的标准库
跨平台编译

# Go 思维

全面简单
- 简洁
- 类型推断
- GC
- 25个 keyword
- package
正交组合
- interface 与其实现之间无显示关联
- 通过组合架构让程序静态结构
- 垂直组合 (类型组合, type embedding)
- 水平组合, 通过 interface 进行组合
偏好并发
- goroutines
- select
- channels
  - 模块之间解耦
  - 并发锁
  - 消息通信

# 共识协议

POW / POS / DPOS / DBFT / DAG / …

1.5 Badger_ Fast Key-Value DB in Go

Cgo is not Go
RocksDB / BoltDB
LSM trees / B+ trees

1.6 Golang在阿里巴巴调度系统Sigma中的实践

package main

import "fmt"

func main() {
        m := map[int][]byte{
                0xc4200761c0: []byte{228, 189, 160, 229, 165, 189, 229, 147, 135},
                0xc4200761c8: []byte{230, 189, 152, 233, 147, 129, 230, 159, 177},
        }

        for _, v := range m {
                fmt.Printf("%p -> %s\n", &v, v)
        }
}

// Output 1
// 0xc420090020 -> 潘铁柱
// 0xc420090020 -> 你好哇
// 
// Output 2
// 0xc420092020 -> 你好哇
// 0xc420092020 -> 潘铁柱

1.7 罗辑思维Go语言微服务改造实践

1.8 Golang打造下一代互联网-IPFS全解析

2.1 Composition In Go

// Reader is the interface that wraps the basic Read method.
type Reader interface {
        Read(p []byte) (n int, err error)
}

// Writer is the interface that wraps the basic Write method.
type Writer interface {
        Write(p []byte) (n int, err error)
}

// ReadWriter is the interface that groups the basic Read and Write methods.
type ReadWriter interface {
        Reader
        Writer
}

2.3 Bazel build Go

// https://bazel.build

// vgo · golang/go Wiki

// golang/dep: Go dependency management tool

2.4 基于Go-Ethereum构建DPOS机制下的区块链

// Ethereum Project

# 以太坊技术协议，以太坊客户端

Geth (Go)
Parity (Rust)
cpp-ethereum (c++)
ethereumj (java)

# 以太坊核心组件

Solidity, a new language for smart contracts, 智能合约
Web3.js

# 以太坊相关工具组

Truffle
Metamask
mist

# 以太坊外部存储

IPFS (Go)
Swarm (Go)

# 共识机制对比

POW
- 消耗计算力
- 出块速度慢，确认慢
- TPS 极低，10~20
- 确认1分支+
DPOS
- 代理人模式
- 出块速度快，确认块
- TPS 700~1000
- 平均确认1~3秒

2.5 深入CGO编程

// cgo is not Go | Dave Cheney

package main

/*
#include <stdio.h>

static void _(const char* s) {
    printf("%s\n", s);
}
*/
import "C"

func main() {
    bytes := *new([]byte)
    bytes = append(bytes, 72)
    bytes = append(bytes, 101, 121)
    bytes = append(bytes, 32, 230, 159)
    bytes = append(bytes, 177, 33, 33, 33)
    C._(C.CString(string(bytes)))
}

$ go build -x hello.go
WORK=/tmp/go-build276115419
mkdir -p $WORK/b001/
cd /tmp
CGO_LDFLAGS='"-g" "-O2"' /usr/local/go/pkg/tool/linux_amd64/cgo -objdir ./go-build276115419/b001/ -importpath command-line-arguments -- -I ./go-build276115419/b001/ -g -O2 ./hello.go
cd $WORK
gcc -fno-caret-diagnostics -c -x c - || true
gcc -Qunused-arguments -c -x c - || true
gcc -fdebug-prefix-map=a=b -c -x c - || true
gcc -gno-record-gcc-switches -c -x c - || true
cd $WORK/b001
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x001.o -c _cgo_export.c
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x002.o -c hello.cgo2.c
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_main.o -c _cgo_main.c
cd /tmp
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -o ./go-build276115419/b001/_cgo_.o ./go-build276115419/b001/_cgo_main.o ./go-build276115419/b001/_x001.o ./go-build276115419/b001/_x002.o -g -O2
/usr/local/go/pkg/tool/linux_amd64/cgo -dynpackage main -dynimport ./go-build276115419/b001/_cgo_.o -dynout ./go-build276115419/b001/_cgo_import.go
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile runtime/cgo=/usr/local/go/pkg/linux_amd64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
EOF
/usr/local/go/pkg/tool/linux_amd64/compile -o ./go-build276115419/b001/_pkg_.a -trimpath ./go-build276115419/b001 -p main -buildid mh6f74THdmFYkLXlaEeI/mh6f74THdmFYkLXlaEeI -goversion go1.10.1 -D _/tmp -importcfg ./go-build276115419/b001/importcfg -pack -c=4 ./go-build276115419/b001/_cgo_gotypes.go ./go-build276115419/b001/hello.cgo1.go ./go-build276115419/b001/_cgo_import.go
/usr/local/go/pkg/tool/linux_amd64/pack r ./go-build276115419/b001/_pkg_.a ./go-build276115419/b001/_x001.o ./go-build276115419/b001/_x002.o # internal
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/_pkg_.a # internal
cp $WORK/b001/_pkg_.a /home/x/.cache/go-build/5f/5fbfbbfb3d3561c7d7f9b6eb2595c5b5c662c609289b0928d54557b768cf6c31-d # internal
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile command-line-arguments=$WORK/b001/_pkg_.a
packagefile runtime/cgo=/usr/local/go/pkg/linux_amd64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
packagefile internal/race=/usr/local/go/pkg/linux_amd64/internal/race.a
packagefile sync=/usr/local/go/pkg/linux_amd64/sync.a
packagefile runtime/internal/atomic=/usr/local/go/pkg/linux_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/go/pkg/linux_amd64/runtime/internal/sys.a
packagefile sync/atomic=/usr/local/go/pkg/linux_amd64/sync/atomic.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=IDPnrZESvB0RYrGsJZll/mh6f74THdmFYkLXlaEeI/_kgrX-BOL__ZqaG2o79P/IDPnrZESvB0RYrGsJZll -extld=gcc $WORK/b001/_pkg_.a
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out hello
rm -r $WORK/b001/

2.6 Go与虚拟化容器 runV/Kata

// hyperhq/runv: Hypervisor-based Runtime for OCI

# Secure as VM, Fast as Container

package main

import (
        "fmt"
)

func main() {
        arr := []struct{ v int }{{1}, {v: 2}, {3}}
        for _, e := range arr {
                e.v++
        }
        fmt.Printf("%v\n", arr)

        for idx := range arr {
                arr[idx].v++
        }
        fmt.Printf("%v\n", arr)
}

// Output
// [{1} {2} {3}]
// [{2} {3} {4}]

package main

import (
        "fmt"
)

func main() {
        s0 := []int{1, 2, 3, 4}
        s1 := s0[:2]              // len=2, cap=4, [1, 2]
        s2 := append(s1, 5)       // len=3, cap=4, [1, 2, 5]
        s3 := append(s1, 6, 7)    // len=4, cap=4, [1, 2, 6, 7]
        s4 := append(s1, 8, 9, 0) // len=5, cap=8, [1, 2, 8, 9, 0]
        fmt.Println(s0)
        fmt.Println(s1)
        fmt.Println(s2)
        fmt.Println(s3)
        fmt.Println(s4)
}

// Output
// [1 2 6 7]
// [1 2]
// [1 2 6]
// [1 2 6 7]
// [1 2 8 9 0]

2.7 Go toolchain internals and implementation based on arm64

// How does the go build command work ? | Dave Cheney

# Go toolchain overview

A toolchain is a package composed of the compiler and ancillary tools, libraries and runtime for a language which together allow you to build and run code written in that language.

gc: evolved from the Plan 9 toolchain and includes its own compiler, assembler, linker and tools, as well as the Go runtime and standard library.
gccgo: extends the gcc project to support Go.
llgo: built on top of the LLVM compiler infrastructure.

# Go toolchain example

package main

import "fmt"

func main() {
        bytes := *new([]byte)
        bytes = append(bytes, 72)
        bytes = append(bytes, 101, 121)
        bytes = append(bytes, 32, 230, 159)
        bytes = append(bytes, 177, 33, 33, 33)
        fmt.Printf("%s\n", bytes)
}

$ go build -x hello.go
WORK=/tmp/go-build173481643
mkdir -p $WORK/b001/
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile fmt=/usr/local/go/pkg/linux_amd64/fmt.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
EOF
cd /tmp
/usr/local/go/pkg/tool/linux_amd64/compile -o ./go-build173481643/b001/_pkg_.a -trimpath ./go-build173481643/b001 -p main -complete -buildid 8sirXYvMbbfw92k19xYW/8sirXYvMbbfw92k19xYW -goversion go1.10.1 -D _/tmp -importcfg ./go-build173481643/b001/importcfg -pack -c=4 ./hello.go
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/_pkg_.a # internal
cp $WORK/b001/_pkg_.a /home/x/.cache/go-build/30/30c2c39b4791f0cd3ff6f312df7ef17888d3eb206e0c11fac21287cff9e0cf3b-d # internal
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile command-line-arguments=$WORK/b001/_pkg_.a
packagefile fmt=/usr/local/go/pkg/linux_amd64/fmt.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
packagefile errors=/usr/local/go/pkg/linux_amd64/errors.a
packagefile io=/usr/local/go/pkg/linux_amd64/io.a
packagefile math=/usr/local/go/pkg/linux_amd64/math.a
packagefile os=/usr/local/go/pkg/linux_amd64/os.a
packagefile reflect=/usr/local/go/pkg/linux_amd64/reflect.a
packagefile strconv=/usr/local/go/pkg/linux_amd64/strconv.a
packagefile sync=/usr/local/go/pkg/linux_amd64/sync.a
packagefile unicode/utf8=/usr/local/go/pkg/linux_amd64/unicode/utf8.a
packagefile runtime/internal/atomic=/usr/local/go/pkg/linux_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/go/pkg/linux_amd64/runtime/internal/sys.a
packagefile sync/atomic=/usr/local/go/pkg/linux_amd64/sync/atomic.a
packagefile internal/cpu=/usr/local/go/pkg/linux_amd64/internal/cpu.a
packagefile internal/poll=/usr/local/go/pkg/linux_amd64/internal/poll.a
packagefile internal/testlog=/usr/local/go/pkg/linux_amd64/internal/testlog.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile time=/usr/local/go/pkg/linux_amd64/time.a
packagefile unicode=/usr/local/go/pkg/linux_amd64/unicode.a
packagefile internal/race=/usr/local/go/pkg/linux_amd64/internal/race.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=Vbu61g9AuYzF2ufqINOR/8sirXYvMbbfw92k19xYW/iWFW984BSmpZp9MnLdjK/Vbu61g9AuYzF2ufqINOR -extld=gcc $WORK/b001/_pkg_.a
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out hello
rm -r $WORK/b001/

package main

/*
 #include <stdio.h>

 static void _(const char* s) {
     printf("%s\n", s);
 }
*/
import "C"

func main() {
        bytes := *new([]byte)
        bytes = append(bytes, 72)
        bytes = append(bytes, 101, 121)
        bytes = append(bytes, 32, 230, 159)
        bytes = append(bytes, 177, 33, 33, 33)
        C._(C.CString(string(bytes)))
}

$ go build -x hello.go
WORK=/tmp/go-build142064459
mkdir -p $WORK/b001/
cd /tmp
CGO_LDFLAGS='"-g" "-O2"' /usr/local/go/pkg/tool/linux_amd64/cgo -objdir ./go-build142064459/b001/ -importpath command-line-arguments -- -I ./go-build142064459/b001/ -g -O2 ./hello.go
cd $WORK
gcc -fno-caret-diagnostics -c -x c - || true
gcc -Qunused-arguments -c -x c - || true
gcc -fdebug-prefix-map=a=b -c -x c - || true
gcc -gno-record-gcc-switches -c -x c - || true
cd $WORK/b001
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x001.o -c _cgo_export.c
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x002.o -c hello.cgo2.c
gcc -I /tmp -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_main.o -c _cgo_main.c
cd /tmp
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -o ./go-build142064459/b001/_cgo_.o ./go-build142064459/b001/_cgo_main.o ./go-build142064459/b001/_x001.o ./go-build142064459/b001/_x002.o -g -O2
/usr/local/go/pkg/tool/linux_amd64/cgo -dynpackage main -dynimport ./go-build142064459/b001/_cgo_.o -dynout ./go-build142064459/b001/_cgo_import.go
cat >$WORK/b001/importcfg << 'EOF' # internal
# import config
packagefile runtime/cgo=/usr/local/go/pkg/linux_amd64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
EOF
/usr/local/go/pkg/tool/linux_amd64/compile -o ./go-build142064459/b001/_pkg_.a -trimpath ./go-build142064459/b001 -p main -buildid WZbBGQxOraNbiT0WofcS/WZbBGQxOraNbiT0WofcS -goversion go1.10.1 -D _/tmp -importcfg ./go-build142064459/b001/importcfg -pack -c=4 ./go-build142064459/b001/_cgo_gotypes.go ./go-build142064459/b001/hello.cgo1.go ./go-build142064459/b001/_cgo_import.go
/usr/local/go/pkg/tool/linux_amd64/pack r ./go-build142064459/b001/_pkg_.a ./go-build142064459/b001/_x001.o ./go-build142064459/b001/_x002.o # internal
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/_pkg_.a # internal
cp $WORK/b001/_pkg_.a /home/x/.cache/go-build/5b/5bedc3923cfc8c06c00e1d8446bf5dd68c35f112d221a1589a8e4ec2fa4b9b11-d # internal
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile command-line-arguments=$WORK/b001/_pkg_.a
packagefile runtime/cgo=/usr/local/go/pkg/linux_amd64/runtime/cgo.a
packagefile syscall=/usr/local/go/pkg/linux_amd64/syscall.a
packagefile runtime=/usr/local/go/pkg/linux_amd64/runtime.a
packagefile internal/race=/usr/local/go/pkg/linux_amd64/internal/race.a
packagefile sync=/usr/local/go/pkg/linux_amd64/sync.a
packagefile runtime/internal/atomic=/usr/local/go/pkg/linux_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/go/pkg/linux_amd64/runtime/internal/sys.a
packagefile sync/atomic=/usr/local/go/pkg/linux_amd64/sync/atomic.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=4aUladWsuR7j-eqT9vPE/WZbBGQxOraNbiT0WofcS/8tvmjI0-hde6HLKu2J8F/4aUladWsuR7j-eqT9vPE -extld=gcc $WORK/b001/_pkg_.a
/usr/local/go/pkg/tool/linux_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out hello
rm -r $WORK/b001/

# Go toolchian overflow

$ go tool
addr2line
asm
buildid
cgo
compile
cover
dist
doc
fix
link
nm
objdump
pack
pprof
test2json
tour
trace
vet

*.go – compile –> $WORK/b001/pkg.a
*.s – asm –> *.o – pack –> runtime=$WORK/b002/pkg.a
internal/bytealg=$WORK/b003/pkg.a
. . .

*.a – link –> hello