$T_{r+1 }$ in the expansion
$\left[ax^2 - \left(\frac{1}{bx}\right)\right]^{11} = ^{11}C_{r} \left(ax^{2}\right)^{11-r} \left(\frac{1}{bx}\right)^{r} $
$= ^{11}C_{r} \left(a\right)^{11-r} \left(b\right)^{-r} \left(x\right)^{22-2r -r} $
For the Coefficient of $x^{7} $ , we have
22 - 3r = 7 $\Rightarrow$ r = 5
$\therefore $ Coefficient of $x^7$
$ = ^{11}C_{5} \left(a\right)^{6} \left(b\right)^{-5}\,\,\,...(1)$
Again $ T_{r+1}$ in the expansion
$ \left[ax - \frac{1}{bx^{2}}\right]^{11} =^{11}C_{r} \left(ax^{2}\right)^{11-r} \left(- \frac{1}{bx^{2}}\right)^{r}$
$ = ^{11}C_{r} \left(a\right)^{11-r} \left(-1\right)^{r} \times\left(b\right)^{-r} \left(x\right)^{-2r} \left(x\right)^{11-r} $
For the Coefficient of $x^{-7}$ , we have
Now 11 - 3r = - 7 $\Rightarrow$ 3r = 18 $\Rightarrow$ r = 6
$\therefore$ Coefficient of $x^{-7}$
$ = {^{11}C_{6}} \, a^5 \times 1 \times (b)^{-6}$
$\therefore $ Coefficient of $x^{7}$ = Coefficient of $x^{-7}$
$ \Rightarrow {^{11}C_5}(a)^6 (b)^{-5} = {^{11}C_6}a^5 \times (b)^{-6} $
$\Rightarrow \, ab = 1$.