hadoop mapreduce 정리

클래스 관점

mapper continer에서
  InputFormat
->RecordReader
->Mapper
->RecordWriter(Partitioner)
->MapOutputBuffer<MapOutputCollector>
->spillAndSort - <partition,key> 기준으로
->SortComparator
->Combiner

reducer container 에서
  Shuffle
->ShuffleScheduler(taskId)
->Fetcher

  merge(SortComparator)
->GroupCompartor
->Reducer
->OutputFormat

---

데이터 관점

InputFormat -> [(k,v),(k,v)...], [(k,v)...] 
Mapper -> {(k,v),(k,v)...}, {(k,v)...} - 순서없는 키
RecordWriter -> [(k,v),(k,v)...], [(k,v)...] - 순서있는 키 , SortComparator
Combiner -> [(k,v),(k,v)...], [(k,v)...] - 순서있는 키

merge -> [(k,v),(k,v)...] - SortComparator
GroupCompartor -> [(k,[v,v...])...] - 
Reducer -> [(k,v),(k,v)...]
OutputFormat

---

InputFormat =
-List<InputSplit> getSplits(JobContext) -client에서 실행, 데이터를 JobContext 에서 읽어서 split 객체 들로 구분, InputSplit(getLength,getLocations,getLocationInfo)
-RecordReader<K, V> createRecordReader(InputSplit, TaskAttemptContext) - node(nodemanager, tasktarcker) InputSplit 에서 Key, value 로 parsing
-LineRecordReader - line의 시작을 가지고 있는 Mapper가 모든 줄을 다 읽고 블록의 시작이 line의 시작이 아니라면 해당 line을 skip 한다.

Mapper
-map

SortComparator - 
-compare

Partitioner
-int getPartition(K2 key, V2 value, int numReduceTasks) - 원하는 reducer 로 분배
-Reducer 한개면 등록한 Partitioner 가 아닌 내부 0 돌려주는 Partitioner 사용

Combiner
-reduce

GroupCompartor - Reducer 의 입력 구분
-compare

Reducer
-reduce


OutputFormat
-RecordWriter<K, V> getRecordWriter(TaskAttemptContext) - Reducer 에게 줄 RecordWriter, RecordWriter(write,close)